2024-02-10T09:03:57.347 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T09:03:57.347 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} --verbose --name yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/orig.config.yaml 2024-02-10T09:03:57.393 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T09:03:57.478 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} email: null first_in_suite: false job_id: '7555369' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(PG_AVAILABILITY\) sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMtRBaNzAsNSb86BThiggZUUAjJPohJ6V4KnVpQc+Lo04N4G5wJ15hooHO5/LKSP69dZL6FCHm/rhQRPDhnniQ= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPThGYEcCYaN+tjd/oImrW4n0Is4gHrAHAeVaySmDdN+HDOZdhSvVh7LPgYKkYS/07ZgDwP79CZmJr2itqi1dD8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - cephadm: add_mons_via_daemon_add: true allow_ptrace: false avoid_pacific_features: true cephadm_branch: v15.2.0 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v15.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw r z --placement=2 --port=8000 - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph config set mgr container_image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.r.z - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_21:34:44 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T09:03:57.479 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-10T09:03:57.479 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-10T09:03:57.479 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T09:03:57.481 INFO:teuthology.task.internal:Checking packages... 2024-02-10T09:03:57.505 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-10T09:03:57.505 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T09:03:57.505 INFO:teuthology.packaging:ref: None 2024-02-10T09:03:57.505 INFO:teuthology.packaging:tag: None 2024-02-10T09:03:57.505 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T09:03:57.505 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:03:57.506 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T09:03:57.658 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-10T09:03:57.659 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T09:03:57.666 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T09:03:57.666 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T09:03:57.726 INFO:teuthology.task.internal:Saving configuration 2024-02-10T09:03:57.743 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T09:03:57.749 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:03:57.774 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:47.029875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMtRBaNzAsNSb86BThiggZUUAjJPohJ6V4KnVpQc+Lo04N4G5wJ15hooHO5/LKSP69dZL6FCHm/rhQRPDhnniQ='} 2024-02-10T09:03:57.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:47.028807', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPThGYEcCYaN+tjd/oImrW4n0Is4gHrAHAeVaySmDdN+HDOZdhSvVh7LPgYKkYS/07ZgDwP79CZmJr2itqi1dD8='} 2024-02-10T09:03:57.799 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T09:03:57.805 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T09:03:57.805 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-10T09:03:57.805 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T09:03:57.893 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd6bdbf20d0>, signals=[15]) 2024-02-10T09:03:57.894 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T09:03:57.899 INFO:teuthology.task.internal:Opening connections... 2024-02-10T09:03:57.899 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-02-10T09:03:57.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:03:57.979 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-02-10T09:03:57.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:03:58.049 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T09:03:58.056 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-02-10T09:03:58.079 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-02-10T09:03:58.079 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T09:03:58.141 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-02-10T09:03:58.142 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T09:03:58.142 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T09:03:58.142 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T09:03:58.142 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-02-10T09:03:58.169 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-02-10T09:03:58.191 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-02-10T09:03:58.191 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="8" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="8" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el8" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T09:03:58.253 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-02-10T09:03:58.254 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T09:03:58.254 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T09:03:58.254 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T09:03:58.254 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-02-10T09:03:58.277 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T09:03:58.286 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T09:03:58.309 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T09:03:58.309 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-02-10T09:03:58.312 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-02-10T09:03:58.332 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T09:03:58.338 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T09:03:58.338 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-02-10T09:03:58.373 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-02-10T09:03:58.400 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T09:03:58.454 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T09:03:58.468 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T09:03:58.468 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T09:03:58.468 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T09:03:58.469 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T09:03:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:03:58.489 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-02-10T09:03:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:03:58.490 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-02-10T09:03:58.550 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-489.el8.x86_64 2024-02-10T09:03:58.550 INFO:teuthology.task.kernel:Running kernel on smithi153: 4.18.0-489.el8.x86_64 2024-02-10T09:03:58.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-02-10T09:03:58.552 INFO:teuthology.orchestra.run.smithi204.stdout:4.18.0-489.el8.x86_64 2024-02-10T09:03:58.552 INFO:teuthology.task.kernel:Running kernel on smithi204: 4.18.0-489.el8.x86_64 2024-02-10T09:03:58.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-02-10T09:04:03.087 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - AppStream 14 MB/s | 27 MB 00:01 2024-02-10T09:04:05.028 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - AppStream 7.0 MB/s | 27 MB 00:03 2024-02-10T09:04:08.974 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - BaseOS 9.2 MB/s | 10 MB 00:01 2024-02-10T09:04:10.982 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2024-02-10T09:04:10.986 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - BaseOS 8.7 MB/s | 10 MB 00:01 2024-02-10T09:04:11.583 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2024-02-10T09:04:11.943 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-02-10T09:04:12.635 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 27 MB/s | 16 MB 00:00 2024-02-10T09:04:12.803 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - Extras 85 kB/s | 18 kB 00:00 2024-02-10T09:04:13.129 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2024-02-10T09:04:13.372 INFO:teuthology.orchestra.run.smithi204.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-10T09:04:16.410 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 365 kB/s | 24 kB 00:00 2024-02-10T09:04:17.984 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 16 MB 00:04 2024-02-10T09:04:19.349 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T09:04:19.349 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T09:04:19.488 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-02-10T09:04:19.489 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-02-10T09:04:19.489 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-02-10T09:04:19.489 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-02-10T09:04:19.489 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-02-10T09:04:19.489 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:Install 3 Packages 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:04:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 90 M 2024-02-10T09:04:19.491 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 96 M 2024-02-10T09:04:19.491 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-02-10T09:04:20.711 INFO:teuthology.orchestra.run.smithi153.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-02-10T09:04:21.707 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 411 kB/s | 24 kB 00:00 2024-02-10T09:04:21.728 INFO:teuthology.orchestra.run.smithi153.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-02-10T09:04:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 17 MB/s | 44 MB 00:02 2024-02-10T09:04:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:04:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:Total 32 MB/s | 90 MB 00:02 2024-02-10T09:04:22.976 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-02-10T09:04:23.424 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-02-10T09:04:23.425 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-02-10T09:04:24.678 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-02-10T09:04:24.704 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T09:04:24.704 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T09:04:24.709 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-02-10T09:04:24.827 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T09:04:24.828 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout:Install 3 Packages 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 90 M 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 96 M 2024-02-10T09:04:24.829 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-02-10T09:04:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-02-10T09:04:27.008 INFO:teuthology.orchestra.run.smithi204.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2024-02-10T09:04:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-02-10T09:04:27.583 INFO:teuthology.orchestra.run.smithi204.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 16 MB/s | 44 MB 00:02 2024-02-10T09:04:27.583 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:04:27.584 INFO:teuthology.orchestra.run.smithi204.stdout:Total 33 MB/s | 90 MB 00:02 2024-02-10T09:04:28.018 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:28.270 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-02-10T09:04:28.676 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-02-10T09:04:28.676 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-02-10T09:04:29.924 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-02-10T09:04:29.940 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-02-10T09:04:31.607 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:31.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-02-10T09:04:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:35.440 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:35.734 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:36.877 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:37.270 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:40.707 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:57.816 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:58.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:58.974 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:58.974 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:04:58.974 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-02-10T09:04:58.974 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T09:04:58.974 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T09:04:58.974 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:04:58.975 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-02-10T09:04:59.135 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:04:59.357 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:04:59.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:04:59.357 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T09:04:59.357 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T09:04:59.357 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T09:04:59.358 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T09:04:59.358 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T09:04:59.358 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:04:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:04:59.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:04:59.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.02 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 129.el8 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T09:04:59.603 INFO:teuthology.orchestra.run.smithi153.stdout:Group : System Environment/Base 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 9265586 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Relocations : (not relocatable) 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : CentOS BuildSystem 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-02-10T09:04:59.604 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:04:59.605 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T09:04:59.605 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T09:04:59.605 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T09:04:59.605 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-02-10T09:04:59.605 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:04:59.605 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T09:04:59.606 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T09:04:59.606 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 4.18.0-536.el8.x86_64 2024-02-10T09:04:59.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T09:05:00.960 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-02-10T09:05:02.910 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:05:02.935 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:05:03.048 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-02-10T09:05:03.051 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-02-10T09:05:03.077 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.t45CeFgrZ8 2024-02-10T09:05:03.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.t45CeFgrZ8 2024-02-10T09:05:03.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.t45CeFgrZ8 2024-02-10T09:05:03.297 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.t45CeFgrZ8 is 6KB 2024-02-10T09:05:03.345 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.t45CeFgrZ8 2024-02-10T09:05:03.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T09:05:03.430 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T09:05:03.430 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T09:05:03.430 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T09:05:03.430 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T09:05:03.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T09:05:03.584 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:05:03.584 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:05:03.584 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:05:03.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-02-10T09:05:04.078 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:05:04.078 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:05:04.079 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-02-10T09:05:04.079 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T09:05:04.079 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T09:05:04.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:05:04.079 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-02-10T09:05:04.214 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:05:04.431 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:05:04.431 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:05:04.431 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T09:05:04.431 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T09:05:04.431 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T09:05:04.431 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T09:05:04.431 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T09:05:04.431 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:05:04.642 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:05:04.642 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:05:04.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-02-10T09:05:04.676 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-02-10T09:05:04.676 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.02 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 129.el8 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Group : System Environment/Base 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 9265586 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Relocations : (not relocatable) 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : CentOS BuildSystem 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-02-10T09:05:04.677 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:05:04.678 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T09:05:04.679 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T09:05:04.679 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 4.18.0-536.el8.x86_64 2024-02-10T09:05:04.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T09:05:06.127 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-02-10T09:05:08.218 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-02-10T09:05:08.220 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-02-10T09:05:08.244 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.7pej6SENTJ 2024-02-10T09:05:08.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7pej6SENTJ 2024-02-10T09:05:08.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.7pej6SENTJ 2024-02-10T09:05:08.461 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.7pej6SENTJ is 6KB 2024-02-10T09:05:08.508 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.7pej6SENTJ 2024-02-10T09:05:08.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T09:05:08.600 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T09:05:08.600 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T09:05:08.600 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T09:05:08.600 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T09:05:08.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T09:05:08.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-02-10T09:05:33.644 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T09:05:33.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:05:33.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:38.748 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T09:05:38.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-02-10T09:05:38.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:52.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-02-10T09:05:57.312 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-02-10T09:06:01.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:06:01.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:04.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-02-10T09:06:06.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-02-10T09:06:06.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:09.376 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-02-10T09:06:16.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:06:16.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:21.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-02-10T09:06:21.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:34.688 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-02-10T09:06:49.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:06:49.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:50.154 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-02-10T09:06:50.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:06:50.429 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-536.el8.x86_64"... 2024-02-10T09:06:50.429 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-02-10T09:06:50.446 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-536.el8.x86_64 2024-02-10T09:06:50.446 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T09:06:50.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T09:06:50.447 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T09:06:51.447 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T09:06:51.448 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T09:06:51.529 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-02-10T09:06:51.545 DEBUG:teuthology.parallel:result is None 2024-02-10T09:06:53.227 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-02-10T09:06:53.578 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-02-10T09:06:53.578 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-536.el8.x86_64"... 2024-02-10T09:06:53.578 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-02-10T09:06:53.595 INFO:teuthology.orchestra.run.smithi204.stdout:4.18.0-536.el8.x86_64 2024-02-10T09:06:53.595 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T09:06:53.595 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T09:06:53.595 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T09:06:54.596 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T09:06:54.596 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T09:06:54.655 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-02-10T09:06:54.671 DEBUG:teuthology.parallel:result is None 2024-02-10T09:06:54.671 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T09:06:54.679 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T09:06:54.679 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T09:06:54.683 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T09:06:54.706 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T09:06:54.724 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T09:06:54.773 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T09:06:54.773 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T09:06:54.795 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T09:06:55.035 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T09:06:55.045 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T09:06:55.045 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T09:06:55.083 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T09:06:55.113 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:55.123 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:55.687 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:55.823 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:55.827 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T09:06:55.835 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T09:06:55.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T09:06:55.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T09:06:56.322 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T09:06:56.332 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T09:06:56.332 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T09:06:56.344 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T09:06:56.361 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:56.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:56.635 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:56.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:56.726 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:06:56.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T09:06:56.797 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:56.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:56.890 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:56.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:57.017 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:06:57.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T09:06:57.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-02-10T09:06:57.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-02-10T09:06:57.143 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:06:57.198 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:06:57.509 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T09:06:57.518 INFO:teuthology.task.internal:Starting timer... 2024-02-10T09:06:57.518 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T09:06:57.567 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T09:06:57.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-02-10T09:06:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-02-10T09:06:57.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-02-10T09:06:57.752 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-02-10T09:06:57.754 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T09:06:57.754 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-02-10T09:06:57.776 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-02-10T09:06:57.777 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-02-10T09:06:57.830 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-02-10T09:06:57.830 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-02-10T09:06:57.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:06:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:06:57.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:06:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:06:57.888 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T09:06:57.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-02-10T09:06:57.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-02-10T09:06:57.954 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T09:06:57.969 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T09:06:57.990 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T09:06:57.991 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T09:15:04.121 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-02-10T09:15:04.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:15:04.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:15:04.199 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-02-10T09:15:04.270 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-02-10T09:15:04.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-02-10T09:15:04.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:15:04.341 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-02-10T09:15:04.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-02-10T09:15:04.410 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T09:15:04.427 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T09:15:04.428 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T09:15:04.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:15:04.431 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T09:15:04.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:15:04.459 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T09:15:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T09:15:04.481 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T09:15:04.494 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T09:15:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-02-10T09:15:04.546 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-02-10T09:15:04.560 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T09:15:04.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-02-10T09:15:04.573 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T09:15:04.574 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-02-10T09:15:04.588 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T09:15:04.601 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T09:15:04.632 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2024-02-10T09:15:04.636 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:15:04.636 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-02-10T09:15:04.636 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.636 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.636 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.636 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.680 INFO:teuthology.orchestra.run.smithi204.stderr:bash: ntpq: command not found 2024-02-10T09:15:04.684 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:15:04.684 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-02-10T09:15:04.684 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.684 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.684 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.684 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:15:04.685 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T09:15:04.695 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T09:15:04.695 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi153.front.sepia.ceph.com 2024-02-10T09:15:04.696 DEBUG:teuthology.orchestra.run.smithi153:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T09:15:04.696 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi204.front.sepia.ceph.com 2024-02-10T09:15:04.696 DEBUG:teuthology.orchestra.run.smithi204:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T09:15:05.473 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 09:13:35 AM UTC. 2024-02-10T09:15:05.511 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:20 ago on Sat 10 Feb 2024 09:13:45 AM UTC. 2024-02-10T09:15:07.429 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout:Resetting modules: 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout: container-tools 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-02-10T09:15:07.430 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:15:07.480 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-02-10T09:15:07.481 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout:Resetting modules: 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout: container-tools 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-02-10T09:15:07.482 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:15:07.685 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-02-10T09:15:07.733 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-02-10T09:15:08.542 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:23 ago on Sat 10 Feb 2024 09:13:45 AM UTC. 2024-02-10T09:15:08.579 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 09:13:35 AM UTC. 2024-02-10T09:15:10.474 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout:=========================================================================================== 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout:=========================================================================================== 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: conmon x86_64 3:2.1.10-1.module_el8+804+f131391c appstream 58 k 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: container-selinux noarch 2:2.227.0-1.module_el8+804+f131391c appstream 70 k 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: containernetworking-plugins x86_64 1:1.4.0-1.module_el8+804+f131391c appstream 22 M 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: containers-common x86_64 2:1-58.module_el8+804+f131391c appstream 148 k 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: criu x86_64 3.18-4.module_el8+804+f131391c appstream 564 k 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-overlayfs x86_64 1.13-1.module_el8+804+f131391c appstream 70 k 2024-02-10T09:15:10.477 INFO:teuthology.orchestra.run.smithi204.stdout: libslirp x86_64 4.4.0-1.module_el8+804+f131391c appstream 70 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: runc x86_64 1:1.1.11-1.module_el8+804+f131391c appstream 3.1 M 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: slirp4netns x86_64 1.2.2-1.module_el8+804+f131391c appstream 56 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout:Installing group/module packages: 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T09:15:10.478 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout: criu-libs x86_64 3.18-4.module_el8+804+f131391c appstream 39 k 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:Installing module profiles: 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout: container-tools/common 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:Enabling module streams: 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout: container-tools rhel8 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:=========================================================================================== 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 9 Packages 2024-02-10T09:15:10.479 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:15:10.480 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 54 M 2024-02-10T09:15:10.480 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-02-10T09:15:10.534 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-02-10T09:15:10.537 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-02-10T09:15:10.537 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-02-10T09:15:10.537 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-02-10T09:15:10.537 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-02-10T09:15:10.537 INFO:teuthology.orchestra.run.smithi153.stdout: conmon x86_64 3:2.1.10-1.module_el8+804+f131391c appstream 58 k 2024-02-10T09:15:10.537 INFO:teuthology.orchestra.run.smithi153.stdout: container-selinux noarch 2:2.227.0-1.module_el8+804+f131391c appstream 70 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: containernetworking-plugins x86_64 1:1.4.0-1.module_el8+804+f131391c appstream 22 M 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: containers-common x86_64 2:1-58.module_el8+804+f131391c appstream 148 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: criu x86_64 3.18-4.module_el8+804+f131391c appstream 564 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-overlayfs x86_64 1.13-1.module_el8+804+f131391c appstream 70 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: libslirp x86_64 4.4.0-1.module_el8+804+f131391c appstream 70 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: runc x86_64 1:1.1.11-1.module_el8+804+f131391c appstream 3.1 M 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: slirp4netns x86_64 1.2.2-1.module_el8+804+f131391c appstream 56 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout:Installing group/module packages: 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T09:15:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: criu-libs x86_64 3.18-4.module_el8+804+f131391c appstream 39 k 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout:Installing module profiles: 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: container-tools/common 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling module streams: 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: container-tools rhel8 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:15:10.539 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-02-10T09:15:10.540 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-02-10T09:15:10.540 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-02-10T09:15:10.540 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 9 Packages 2024-02-10T09:15:10.540 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:15:10.540 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 54 M 2024-02-10T09:15:10.540 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-02-10T09:15:11.081 INFO:teuthology.orchestra.run.smithi204.stdout:(1/22): aardvark-dns-1.9.0-1.module_el8+804+f13 1.9 MB/s | 1.0 MB 00:00 2024-02-10T09:15:11.106 INFO:teuthology.orchestra.run.smithi204.stdout:(2/22): cockpit-podman-82-1.module_el8+804+f131 1.3 MB/s | 759 kB 00:00 2024-02-10T09:15:11.173 INFO:teuthology.orchestra.run.smithi204.stdout:(3/22): criu-libs-3.18-4.module_el8+804+f131391 427 kB/s | 39 kB 00:00 2024-02-10T09:15:11.424 INFO:teuthology.orchestra.run.smithi204.stdout:(4/22): crun-1.12-1.module_el8+804+f131391c.x86 784 kB/s | 248 kB 00:00 2024-02-10T09:15:11.604 INFO:teuthology.orchestra.run.smithi153.stdout:(1/22): cockpit-podman-82-1.module_el8+804+f131 1.3 MB/s | 759 kB 00:00 2024-02-10T09:15:11.632 INFO:teuthology.orchestra.run.smithi204.stdout:(5/22): python3-podman-4.8.2-1.module_el8+804+f 861 kB/s | 179 kB 00:00 2024-02-10T09:15:11.654 INFO:teuthology.orchestra.run.smithi153.stdout:(2/22): aardvark-dns-1.9.0-1.module_el8+804+f13 1.6 MB/s | 1.0 MB 00:00 2024-02-10T09:15:11.671 INFO:teuthology.orchestra.run.smithi153.stdout:(3/22): criu-libs-3.18-4.module_el8+804+f131391 588 kB/s | 39 kB 00:00 2024-02-10T09:15:11.707 INFO:teuthology.orchestra.run.smithi204.stdout:(6/22): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-02-10T09:15:11.772 INFO:teuthology.orchestra.run.smithi153.stdout:(4/22): crun-1.12-1.module_el8+804+f131391c.x86 2.1 MB/s | 248 kB 00:00 2024-02-10T09:15:12.024 INFO:teuthology.orchestra.run.smithi204.stdout:(7/22): python3-pyxdg-0.25-16.el8.noarch.rpm 296 kB/s | 94 kB 00:00 2024-02-10T09:15:12.039 INFO:teuthology.orchestra.run.smithi153.stdout:(5/22): python3-podman-4.8.2-1.module_el8+804+f 673 kB/s | 179 kB 00:00 2024-02-10T09:15:12.139 INFO:teuthology.orchestra.run.smithi153.stdout:(6/22): python3-pytoml-0.1.14-5.git7dea353.el8. 247 kB/s | 25 kB 00:00 2024-02-10T09:15:12.281 INFO:teuthology.orchestra.run.smithi153.stdout:(7/22): python3-pyxdg-0.25-16.el8.noarch.rpm 663 kB/s | 94 kB 00:00 2024-02-10T09:15:13.006 INFO:teuthology.orchestra.run.smithi153.stdout:(8/22): netavark-1.9.0-1.module_el8+804+f131391 3.0 MB/s | 4.0 MB 00:01 2024-02-10T09:15:13.258 INFO:teuthology.orchestra.run.smithi204.stdout:(8/22): netavark-1.9.0-1.module_el8+804+f131391 1.9 MB/s | 4.0 MB 00:02 2024-02-10T09:15:13.864 INFO:teuthology.orchestra.run.smithi153.stdout:(9/22): toolbox-0.0.99.4-8.module_el8+804+f1313 2.9 MB/s | 2.5 MB 00:00 2024-02-10T09:15:13.956 INFO:teuthology.orchestra.run.smithi153.stdout:(10/22): udica-0.2.6-20.module_el8+804+f131391c 576 kB/s | 53 kB 00:00 2024-02-10T09:15:14.032 INFO:teuthology.orchestra.run.smithi153.stdout:(11/22): yajl-2.1.0-12.el8.x86_64.rpm 546 kB/s | 41 kB 00:00 2024-02-10T09:15:14.098 INFO:teuthology.orchestra.run.smithi153.stdout:(12/22): conmon-2.1.10-1.module_el8+804+f131391 870 kB/s | 58 kB 00:00 2024-02-10T09:15:14.182 INFO:teuthology.orchestra.run.smithi153.stdout:(13/22): container-selinux-2.227.0-1.module_el8 845 kB/s | 70 kB 00:00 2024-02-10T09:15:14.442 INFO:teuthology.orchestra.run.smithi204.stdout:(9/22): toolbox-0.0.99.4-8.module_el8+804+f1313 2.1 MB/s | 2.5 MB 00:01 2024-02-10T09:15:14.517 INFO:teuthology.orchestra.run.smithi204.stdout:(10/22): udica-0.2.6-20.module_el8+804+f131391c 704 kB/s | 53 kB 00:00 2024-02-10T09:15:14.592 INFO:teuthology.orchestra.run.smithi204.stdout:(11/22): yajl-2.1.0-12.el8.x86_64.rpm 546 kB/s | 41 kB 00:00 2024-02-10T09:15:14.676 INFO:teuthology.orchestra.run.smithi204.stdout:(12/22): conmon-2.1.10-1.module_el8+804+f131391 696 kB/s | 58 kB 00:00 2024-02-10T09:15:14.768 INFO:teuthology.orchestra.run.smithi204.stdout:(13/22): container-selinux-2.227.0-1.module_el8 768 kB/s | 70 kB 00:00 2024-02-10T09:15:14.983 INFO:teuthology.orchestra.run.smithi153.stdout:(14/22): buildah-1.34.0-1.module_el8+804+f13139 2.4 MB/s | 9.7 MB 00:03 2024-02-10T09:15:15.083 INFO:teuthology.orchestra.run.smithi153.stdout:(15/22): skopeo-1.14.0-2.module_el8+804+f131391 3.1 MB/s | 8.8 MB 00:02 2024-02-10T09:15:15.158 INFO:teuthology.orchestra.run.smithi153.stdout:(16/22): containers-common-1-58.module_el8+804+ 844 kB/s | 148 kB 00:00 2024-02-10T09:15:15.233 INFO:teuthology.orchestra.run.smithi153.stdout:(17/22): fuse-overlayfs-1.13-1.module_el8+804+f 935 kB/s | 70 kB 00:00 2024-02-10T09:15:15.268 INFO:teuthology.orchestra.run.smithi204.stdout:(14/22): buildah-1.34.0-1.module_el8+804+f13139 2.0 MB/s | 9.7 MB 00:04 2024-02-10T09:15:15.300 INFO:teuthology.orchestra.run.smithi153.stdout:(18/22): libslirp-4.4.0-1.module_el8+804+f13139 1.0 MB/s | 70 kB 00:00 2024-02-10T09:15:15.350 INFO:teuthology.orchestra.run.smithi153.stdout:(19/22): criu-3.18-4.module_el8+804+f131391c.x8 2.1 MB/s | 564 kB 00:00 2024-02-10T09:15:15.402 INFO:teuthology.orchestra.run.smithi204.stdout:(15/22): containers-common-1-58.module_el8+804+ 1.1 MB/s | 148 kB 00:00 2024-02-10T09:15:15.417 INFO:teuthology.orchestra.run.smithi153.stdout:(20/22): slirp4netns-1.2.2-1.module_el8+804+f13 839 kB/s | 56 kB 00:00 2024-02-10T09:15:15.969 INFO:teuthology.orchestra.run.smithi204.stdout:(16/22): criu-3.18-4.module_el8+804+f131391c.x8 996 kB/s | 564 kB 00:00 2024-02-10T09:15:16.102 INFO:teuthology.orchestra.run.smithi204.stdout:(17/22): fuse-overlayfs-1.13-1.module_el8+804+f 525 kB/s | 70 kB 00:00 2024-02-10T09:15:16.228 INFO:teuthology.orchestra.run.smithi204.stdout:(18/22): libslirp-4.4.0-1.module_el8+804+f13139 564 kB/s | 70 kB 00:00 2024-02-10T09:15:17.184 INFO:teuthology.orchestra.run.smithi153.stdout:(21/22): runc-1.1.11-1.module_el8+804+f131391c. 1.7 MB/s | 3.1 MB 00:01 2024-02-10T09:15:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:(22/22): containernetworking-plugins-1.4.0-1.mo 4.8 MB/s | 22 MB 00:04 2024-02-10T09:15:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:15:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.5 MB/s | 54 MB 00:08 2024-02-10T09:15:18.944 INFO:teuthology.orchestra.run.smithi204.stdout:(19/22): skopeo-1.14.0-2.module_el8+804+f131391 1.3 MB/s | 8.8 MB 00:06 2024-02-10T09:15:19.028 INFO:teuthology.orchestra.run.smithi204.stdout:(20/22): runc-1.1.11-1.module_el8+804+f131391c. 1.1 MB/s | 3.1 MB 00:02 2024-02-10T09:15:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:(21/22): slirp4netns-1.2.2-1.module_el8+804+f13 516 kB/s | 56 kB 00:00 2024-02-10T09:15:19.230 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-02-10T09:15:19.276 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-02-10T09:15:19.276 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-02-10T09:15:19.755 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-02-10T09:15:19.758 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-02-10T09:15:20.983 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-02-10T09:15:20.995 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f131391 1/1 2024-02-10T09:15:21.036 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f1313 1/31 2024-02-10T09:15:21.257 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : container-selinux-2:2.227.0-1.module_el8+804+f1313 1/31 2024-02-10T09:15:22.195 INFO:teuthology.orchestra.run.smithi204.stdout:(22/22): containernetworking-plugins-1.4.0-1.mo 3.0 MB/s | 22 MB 00:07 2024-02-10T09:15:22.195 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:15:22.195 INFO:teuthology.orchestra.run.smithi204.stdout:Total 4.6 MB/s | 54 MB 00:11 2024-02-10T09:15:22.696 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-02-10T09:15:22.741 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-02-10T09:15:22.741 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-02-10T09:15:23.227 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-02-10T09:15:23.230 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-02-10T09:15:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-02-10T09:15:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f131391 1/1 2024-02-10T09:15:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f1313 1/31 2024-02-10T09:15:24.979 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : container-selinux-2:2.227.0-1.module_el8+804+f1313 1/31 2024-02-10T09:15:38.979 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f1313 1/31 2024-02-10T09:15:39.531 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : criu-3.18-4.module_el8+804+f131391c.x86_64 2/31 2024-02-10T09:15:39.761 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : runc-1:1.1.11-1.module_el8+804+f131391c.x86_64 3/31 2024-02-10T09:15:39.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : criu-libs-3.18-4.module_el8+804+f131391c.x86_64 4/31 2024-02-10T09:15:40.103 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libslirp-4.4.0-1.module_el8+804+f131391c.x86_64 5/31 2024-02-10T09:15:40.347 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : slirp4netns-1.2.2-1.module_el8+804+f131391c.x86_64 6/31 2024-02-10T09:15:40.554 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_ 7/31 2024-02-10T09:15:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_ 7/31 2024-02-10T09:15:42.638 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : containernetworking-plugins-1:1.4.0-1.module_el8+8 8/31 2024-02-10T09:15:42.793 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f1313 1/31 2024-02-10T09:15:42.841 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : yajl-2.1.0-12.el8.x86_64 9/31 2024-02-10T09:15:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 10/31 2024-02-10T09:15:43.275 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : containers-common-2:1-58.module_el8+804+f131391c.x 11/31 2024-02-10T09:15:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : criu-3.18-4.module_el8+804+f131391c.x86_64 2/31 2024-02-10T09:15:43.462 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/31 2024-02-10T09:15:43.591 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : runc-1:1.1.11-1.module_el8+804+f131391c.x86_64 3/31 2024-02-10T09:15:43.737 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : criu-libs-3.18-4.module_el8+804+f131391c.x86_64 4/31 2024-02-10T09:15:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 13/31 2024-02-10T09:15:43.908 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libslirp-4.4.0-1.module_el8+804+f131391c.x86_64 5/31 2024-02-10T09:15:44.128 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : slirp4netns-1.2.2-1.module_el8+804+f131391c.x86_64 6/31 2024-02-10T09:15:44.294 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 14/31 2024-02-10T09:15:44.302 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_ 7/31 2024-02-10T09:15:45.250 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 15/31 2024-02-10T09:15:45.543 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 16/31 2024-02-10T09:15:46.116 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_ 7/31 2024-02-10T09:15:46.367 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : containernetworking-plugins-1:1.4.0-1.module_el8+8 8/31 2024-02-10T09:15:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 17/31 2024-02-10T09:15:46.563 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : yajl-2.1.0-12.el8.x86_64 9/31 2024-02-10T09:15:46.770 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 10/31 2024-02-10T09:15:46.950 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 18/31 2024-02-10T09:15:47.026 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : containers-common-2:1-58.module_el8+804+f131391c.x 11/31 2024-02-10T09:15:47.149 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 19/31 2024-02-10T09:15:47.218 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/31 2024-02-10T09:15:47.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 20/31 2024-02-10T09:15:47.503 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 13/31 2024-02-10T09:15:47.575 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : conmon-3:2.1.10-1.module_el8+804+f131391c.x86_64 21/31 2024-02-10T09:15:47.771 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 22/31 2024-02-10T09:15:47.937 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : containers-common-2:1-49.module_el8.8.0+1254+78119 23/31 2024-02-10T09:15:48.048 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 14/31 2024-02-10T09:15:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : slirp4netns-1.2.0-2.module_el8.7.0+1216+b022c01d.x 24/31 2024-02-10T09:15:48.279 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : runc-1:1.1.4-1.module_el8.7.0+1216+b022c01d.x86_64 25/31 2024-02-10T09:15:48.279 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : container-selinux-2:2.195.1-1.module_el8.8.0+1254+ 26/31 2024-02-10T09:15:48.488 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: container-selinux-2:2.195.1-1.module_el8.8.0+1254+ 26/31 2024-02-10T09:15:48.703 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : criu-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 27/31 2024-02-10T09:15:48.863 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libslirp-4.4.0-1.module_el8.7.0+1216+b022c01d.x86_ 28/31 2024-02-10T09:15:49.027 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 15/31 2024-02-10T09:15:49.062 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : fuse-overlayfs-1.10-1.module_el8.8.0+1254+78119b6e 29/31 2024-02-10T09:15:49.279 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : containernetworking-plugins-1:1.1.1-3.module_el8.7 30/31 2024-02-10T09:15:49.295 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 16/31 2024-02-10T09:15:49.470 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : conmon-3:2.1.5-1.module_el8.8.0+1254+78119b6e.x86_ 31/31 2024-02-10T09:15:50.286 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 17/31 2024-02-10T09:15:50.704 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 18/31 2024-02-10T09:15:50.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 19/31 2024-02-10T09:15:51.092 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 20/31 2024-02-10T09:15:51.332 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : conmon-3:2.1.10-1.module_el8+804+f131391c.x86_64 21/31 2024-02-10T09:15:51.535 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 22/31 2024-02-10T09:15:51.730 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : containers-common-2:1-49.module_el8.8.0+1254+78119 23/31 2024-02-10T09:15:51.903 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : slirp4netns-1.2.0-2.module_el8.7.0+1216+b022c01d.x 24/31 2024-02-10T09:15:52.069 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : runc-1:1.1.4-1.module_el8.7.0+1216+b022c01d.x86_64 25/31 2024-02-10T09:15:52.069 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : container-selinux-2:2.195.1-1.module_el8.8.0+1254+ 26/31 2024-02-10T09:15:52.261 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: container-selinux-2:2.195.1-1.module_el8.8.0+1254+ 26/31 2024-02-10T09:15:52.447 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : criu-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 27/31 2024-02-10T09:15:52.627 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libslirp-4.4.0-1.module_el8.7.0+1216+b022c01d.x86_ 28/31 2024-02-10T09:15:52.805 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : fuse-overlayfs-1.10-1.module_el8.8.0+1254+78119b6e 29/31 2024-02-10T09:15:52.994 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : containernetworking-plugins-1:1.1.1-3.module_el8.7 30/31 2024-02-10T09:15:53.176 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : conmon-3:2.1.5-1.module_el8.8.0+1254+78119b6e.x86_ 31/31 2024-02-10T09:16:08.700 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f1313 31/31 2024-02-10T09:16:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: conmon-3:2.1.5-1.module_el8.8.0+1254+78119b6e.x86_ 31/31 2024-02-10T09:16:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/31 2024-02-10T09:16:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/31 2024-02-10T09:16:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/31 2024-02-10T09:16:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : criu-libs-3.18-4.module_el8+804+f131391c.x86_64 4/31 2024-02-10T09:16:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : conmon-3:2.1.10-1.module_el8+804+f131391c.x86_64 14/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : conmon-3:2.1.5-1.module_el8.8.0+1254+78119b6e.x86_ 15/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : container-selinux-2:2.227.0-1.module_el8+804+f1313 16/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : container-selinux-2:2.195.1-1.module_el8.8.0+1254+ 17/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : containernetworking-plugins-1:1.4.0-1.module_el8+8 18/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module_el8.7 19/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : containers-common-2:1-58.module_el8+804+f131391c.x 20/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : containers-common-2:1-49.module_el8.8.0+1254+78119 21/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : criu-3.18-4.module_el8+804+f131391c.x86_64 22/31 2024-02-10T09:16:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : criu-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 23/31 2024-02-10T09:16:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_ 24/31 2024-02-10T09:16:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-overlayfs-1.10-1.module_el8.8.0+1254+78119b6e 25/31 2024-02-10T09:16:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libslirp-4.4.0-1.module_el8+804+f131391c.x86_64 26/31 2024-02-10T09:16:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libslirp-4.4.0-1.module_el8.7.0+1216+b022c01d.x86_ 27/31 2024-02-10T09:16:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : runc-1:1.1.11-1.module_el8+804+f131391c.x86_64 28/31 2024-02-10T09:16:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : runc-1:1.1.4-1.module_el8.7.0+1216+b022c01d.x86_64 29/31 2024-02-10T09:16:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : slirp4netns-1.2.2-1.module_el8+804+f131391c.x86_64 30/31 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : slirp4netns-1.2.0-2.module_el8.7.0+1216+b022c01d.x 31/31 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: conmon-3:2.1.10-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: container-selinux-2:2.227.0-1.module_el8+804+f131391c.noarch 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: containernetworking-plugins-1:1.4.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: containers-common-2:1-58.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: criu-3.18-4.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.996 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: libslirp-4.4.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: runc-1:1.1.11-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: slirp4netns-1.2.2-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: criu-libs-3.18-4.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.997 INFO:teuthology.orchestra.run.smithi153.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:16:09.998 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-02-10T09:16:10.284 DEBUG:teuthology.parallel:result is None 2024-02-10T09:16:12.324 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: container-selinux-2:2.227.0-1.module_el8+804+f1313 31/31 2024-02-10T09:16:12.806 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: conmon-3:2.1.5-1.module_el8.8.0+1254+78119b6e.x86_ 31/31 2024-02-10T09:16:12.806 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/31 2024-02-10T09:16:12.806 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/31 2024-02-10T09:16:12.806 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/31 2024-02-10T09:16:12.806 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : criu-libs-3.18-4.module_el8+804+f131391c.x86_64 4/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : conmon-3:2.1.10-1.module_el8+804+f131391c.x86_64 14/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : conmon-3:2.1.5-1.module_el8.8.0+1254+78119b6e.x86_ 15/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : container-selinux-2:2.227.0-1.module_el8+804+f1313 16/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : container-selinux-2:2.195.1-1.module_el8.8.0+1254+ 17/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : containernetworking-plugins-1:1.4.0-1.module_el8+8 18/31 2024-02-10T09:16:12.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module_el8.7 19/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : containers-common-2:1-58.module_el8+804+f131391c.x 20/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : containers-common-2:1-49.module_el8.8.0+1254+78119 21/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : criu-3.18-4.module_el8+804+f131391c.x86_64 22/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : criu-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 23/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_ 24/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-overlayfs-1.10-1.module_el8.8.0+1254+78119b6e 25/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libslirp-4.4.0-1.module_el8+804+f131391c.x86_64 26/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libslirp-4.4.0-1.module_el8.7.0+1216+b022c01d.x86_ 27/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : runc-1:1.1.11-1.module_el8+804+f131391c.x86_64 28/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : runc-1:1.1.4-1.module_el8.7.0+1216+b022c01d.x86_64 29/31 2024-02-10T09:16:12.808 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : slirp4netns-1.2.2-1.module_el8+804+f131391c.x86_64 30/31 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : slirp4netns-1.2.0-2.module_el8.7.0+1216+b022c01d.x 31/31 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout: conmon-3:2.1.10-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout: container-selinux-2:2.227.0-1.module_el8+804+f131391c.noarch 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout: containernetworking-plugins-1:1.4.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.511 INFO:teuthology.orchestra.run.smithi204.stdout: containers-common-2:1-58.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: criu-3.18-4.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-overlayfs-1.13-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: libslirp-4.4.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: runc-1:1.1.11-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: slirp4netns-1.2.2-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: criu-libs-3.18-4.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.512 INFO:teuthology.orchestra.run.smithi204.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:16:13.513 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-02-10T09:16:13.802 DEBUG:teuthology.parallel:result is None 2024-02-10T09:16:13.802 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T09:16:13.877 INFO:tasks.cephadm:Config: {'add_mons_via_daemon_add': True, 'allow_ptrace': False, 'avoid_pacific_features': True, 'cephadm_branch': 'v15.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v15.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-10T09:16:13.877 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v15.2.0 2024-02-10T09:16:13.878 INFO:tasks.cephadm:Cluster fsid is 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:13.878 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T09:16:13.878 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.153', 'mon.c': '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]', 'mon.b': '172.21.15.204'} 2024-02-10T09:16:13.878 INFO:tasks.cephadm:First mon is mon.a on smithi153 2024-02-10T09:16:13.878 INFO:tasks.cephadm:First mgr is y 2024-02-10T09:16:13.879 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T09:16:13.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-02-10T09:16:13.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-02-10T09:16:13.941 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v15.2.0)... 2024-02-10T09:16:13.942 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v15.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T09:16:14.268 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-rw-r--. 1 ubuntu ubuntu 151122 Feb 10 09:16 /home/ubuntu/cephtest/cephadm 2024-02-10T09:16:14.269 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v15.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T09:16:14.453 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-rw-r--. 1 ubuntu ubuntu 151122 Feb 10 09:16 /home/ubuntu/cephtest/cephadm 2024-02-10T09:16:14.453 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T09:16:14.476 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T09:16:14.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-02-10T09:16:14.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-02-10T09:16:14.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-02-10T09:16:14.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-02-10T09:16:14.643 INFO:tasks.cephadm:Writing seed config... 2024-02-10T09:16:14.644 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T09:16:14.645 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T09:16:14.646 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:16:14.646 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T09:16:14.677 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 09b43244-c7f5-11ee-95b8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T09:16:14.677 DEBUG:teuthology.orchestra.run.smithi153:mon.a> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T09:16:14.719 DEBUG:teuthology.orchestra.run.smithi153:mgr.y> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T09:16:14.748 INFO:journalctl@ceph.mon.a.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:16:14.749 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T09:16:14.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 -v bootstrap --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.153 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:16:14.776 INFO:journalctl@ceph.mgr.y.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:16:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Verifying podman|docker is present... 2024-02-10T09:16:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Verifying lvm2 is present... 2024-02-10T09:16:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Verifying time synchronization is in place... 2024-02-10T09:16:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-enabled chrony.service 2024-02-10T09:16:14.900 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T09:16:14.900 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-active chrony.service 2024-02-10T09:16:14.905 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout inactive 2024-02-10T09:16:14.905 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-enabled chronyd.service 2024-02-10T09:16:14.910 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout enabled 2024-02-10T09:16:14.910 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-active chronyd.service 2024-02-10T09:16:14.915 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout active 2024-02-10T09:16:14.915 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Unit chronyd.service is enabled and running 2024-02-10T09:16:14.915 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Repeating the final host check... 2024-02-10T09:16:14.916 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:podman|docker (/bin/podman) is present 2024-02-10T09:16:14.916 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:systemctl is present 2024-02-10T09:16:14.916 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:lvcreate is present 2024-02-10T09:16:14.916 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-enabled chrony.service 2024-02-10T09:16:14.920 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T09:16:14.920 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-active chrony.service 2024-02-10T09:16:14.924 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout inactive 2024-02-10T09:16:14.924 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-enabled chronyd.service 2024-02-10T09:16:14.929 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout enabled 2024-02-10T09:16:14.929 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-active chronyd.service 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout active 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Unit chronyd.service is enabled and running 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Host looks OK 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:root:Cluster fsid: 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Acquiring lock 140191482345624 on /run/cephadm/09b43244-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Lock 140191482345624 acquired on /run/cephadm/09b43244-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:16:14.934 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Verifying IP 172.21.15.153 port 3300 ... 2024-02-10T09:16:14.935 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Verifying IP 172.21.15.153 port 6789 ... 2024-02-10T09:16:14.935 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Base mon IP is 172.21.15.153, final addrv is [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-02-10T09:16:14.935 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /sbin/ip route ls 2024-02-10T09:16:14.936 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/sbin/ip:stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.153 metric 100 2024-02-10T09:16:14.936 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/sbin/ip:stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.153 metric 100 2024-02-10T09:16:14.937 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Mon IP 172.21.15.153 is in CIDR network 172.21.0.0/20 2024-02-10T09:16:14.938 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15.2.0 container... 2024-02-10T09:16:14.938 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman pull docker.io/ceph/ceph:v15.2.0 2024-02-10T09:16:16.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stderr Trying to pull docker.io/ceph/ceph:v15.2.0... 2024-02-10T09:16:23.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stderr Getting image source signatures 2024-02-10T09:16:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stderr Copying blob sha256:120f0bd0d532e2094e4c344cda2d32931787ee8c2e5d6b9ddd4698dfe3ef6110 2024-02-10T09:16:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stderr Copying blob sha256:8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07 2024-02-10T09:16:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stderr Copying config sha256:204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1 2024-02-10T09:16:34.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stderr Writing manifest to image destination 2024-02-10T09:16:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/bin/podman:stdout 204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1 2024-02-10T09:16:34.594 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Extracting ceph user uid/gid from container image... 2024-02-10T09:16:34.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '--entrypoint', 'stat', 'docker.io/ceph/ceph:v15.2.0', '-c', '%u %g', '/var/lib/ceph'] 2024-02-10T09:16:34.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 --entrypoint stat docker.io/ceph/ceph:v15.2.0 -c %u %g /var/lib/ceph 2024-02-10T09:16:35.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:stat:stdout 167 167 2024-02-10T09:16:35.640 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Creating initial keys... 2024-02-10T09:16:35.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '--entrypoint', '/usr/bin/ceph-authtool', 'docker.io/ceph/ceph:v15.2.0', '--gen-print-key'] 2024-02-10T09:16:35.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 --entrypoint /usr/bin/ceph-authtool docker.io/ceph/ceph:v15.2.0 --gen-print-key 2024-02-10T09:16:36.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-authtool:stdout AQD0Psdl0B0dARAAT3br68RKTcaYfgup6aS2BA== 2024-02-10T09:16:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '--entrypoint', '/usr/bin/ceph-authtool', 'docker.io/ceph/ceph:v15.2.0', '--gen-print-key'] 2024-02-10T09:16:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 --entrypoint /usr/bin/ceph-authtool docker.io/ceph/ceph:v15.2.0 --gen-print-key 2024-02-10T09:16:37.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-authtool:stdout AQD1PsdlgplgCBAAJeoO1m5KToyyWBRGs2nsZw== 2024-02-10T09:16:37.808 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '--entrypoint', '/usr/bin/ceph-authtool', 'docker.io/ceph/ceph:v15.2.0', '--gen-print-key'] 2024-02-10T09:16:37.808 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 --entrypoint /usr/bin/ceph-authtool docker.io/ceph/ceph:v15.2.0 --gen-print-key 2024-02-10T09:16:38.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-authtool:stdout AQD2PsdlYcytDxAA9sWcu7zYtIxdw7R4su/rBA== 2024-02-10T09:16:38.829 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Creating initial monmap... 2024-02-10T09:16:38.829 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/tmp/ceph-tmp6we_sqnd:/tmp/monmap:z', '--entrypoint', '/usr/bin/monmaptool', 'docker.io/ceph/ceph:v15.2.0', '--create', '--clobber', '--fsid', '09b43244-c7f5-11ee-95b8-87774f69a715', '--addv', 'a', '[v2:172.21.15.153:3300,v1:172.21.15.153:6789]', '/tmp/monmap'] 2024-02-10T09:16:38.829 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /tmp/ceph-tmp6we_sqnd:/tmp/monmap:z --entrypoint /usr/bin/monmaptool docker.io/ceph/ceph:v15.2.0 --create --clobber --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 --addv a [v2:172.21.15.153:3300,v1:172.21.15.153:6789] /tmp/monmap 2024-02-10T09:16:39.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/monmaptool:stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T09:16:39.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/monmaptool:stdout /usr/bin/monmaptool: set fsid to 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:39.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/monmaptool:stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T09:16:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Creating mon... 2024-02-10T09:16:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z', '-v', '/tmp/ceph-tmp_sml7pov:/tmp/keyring:z', '-v', '/tmp/ceph-tmp6we_sqnd:/tmp/monmap:z', '--entrypoint', '/usr/bin/ceph-mon', 'docker.io/ceph/ceph:v15.2.0', '--mkfs', '-i', 'a', '--fsid', '09b43244-c7f5-11ee-95b8-87774f69a715', '-c', '/dev/null', '--monmap', '/tmp/monmap', '--keyring', '/tmp/keyring', '--setuser', 'ceph', '--setgroup', 'ceph', '--default-log-to-file=false', '--default-log-to-stderr=true', '--default-log-stderr-prefix="debug "', '--default-mon-cluster-log-to-file=false', '--default-mon-cluster-log-to-stderr=true'] 2024-02-10T09:16:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp_sml7pov:/tmp/keyring:z -v /tmp/ceph-tmp6we_sqnd:/tmp/monmap:z --entrypoint /usr/bin/ceph-mon docker.io/ceph/ceph:v15.2.0 --mkfs -i a --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix="debug " --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-10T09:16:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.324+0000 7fb0777f66c0 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:16:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.324+0000 7fb0777f66c0 1 imported monmap: 2024-02-10T09:16:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr epoch 0 2024-02-10T09:16:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr last_changed 2024-02-10T09:16:39.264924+0000 2024-02-10T09:16:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr created 2024-02-10T09:16:39.264924+0000 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr min_mon_release 0 (unknown) 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 /usr/bin/ceph-mon: set fsid to 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 set rocksdb option compression = kNoCompression 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 set rocksdb option write_buffer_size = 33554432 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 set rocksdb option compression = kNoCompression 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 0 set rocksdb option write_buffer_size = 33554432 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: RocksDB version: 6.1.2 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Compile date Mar 23 2020 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: DB SUMMARY 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.create_if_missing: 1 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.env: 0x55838876e1a0 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.info_log: 0x558389827aa0 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:16:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.db_log_dir: 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:16:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.write_buffer_manager: 0x55838a3c6120 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:16:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.row_cache: None 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.wal_filter: None 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:16:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Compression algorithms supported: 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kZSTD supported: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:16:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.325+0000 7fb0777f66c0 4 rocksdb: [db/db_impl_open.cc:228] Creating manifest 1 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]: 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.merge_operator: 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:16:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55838974c118) 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr cache_index_and_filter_blocks: 1 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr cache_index_and_filter_blocks_with_high_priority: 1 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr pin_top_level_index_and_filter: 1 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr index_type: 0 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr data_block_index_type: 0 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr hash_index_allow_collision: 1 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr checksum: 1 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr no_block_cache: 0 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_cache: 0x558389786510 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_cache_name: BinnedLRUCache 2024-02-10T09:16:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_cache_options: 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr capacity : 536870912 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr num_shard_bits : 4 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr strict_capacity_limit : 0 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr high_pri_pool_ratio: 0.000 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_cache_compressed: (nil) 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr persistent_cache: (nil) 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_size: 4096 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_size_deviation: 10 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_restart_interval: 16 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr index_block_restart_interval: 1 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr metadata_block_size: 4096 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr partition_filters: 0 2024-02-10T09:16:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr use_delta_encoding: 1 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr whole_key_filtering: 1 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr verify_compression: 0 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr read_amp_bytes_per_bit: 0 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr format_version: 2 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr enable_index_compression: 1 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr block_align: 0 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:16:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.num_levels: 7 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:16:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:16:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:16:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:16:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.326+0000 7fb0777f66c0 4 rocksdb: Options.ttl: 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.327+0000 7fb0777f66c0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.327+0000 7fb0777f66c0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 0 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.329+0000 7fb0777f66c0 4 rocksdb: DB pointer 0x558389821200 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.329+0000 7fb0609b7700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.329+0000 7fb0609b7700 4 rocksdb: [db/db_impl.cc:778] 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** DB Stats ** 2024-02-10T09:16:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** Compaction Stats [default] ** 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** Compaction Stats [default] ** 2024-02-10T09:16:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** Compaction Stats [default] ** 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** Compaction Stats [default] ** 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:16:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.331+0000 7fb0777f66c0 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.331+0000 7fb0777f66c0 4 rocksdb: [db/db_impl.cc:563] Shutdown complete 2024-02-10T09:16:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph-mon:stderr "debug "2024-02-10T09:16:40.331+0000 7fb0777f66c0 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-10T09:16:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:40.965 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl disable ceph.target 2024-02-10T09:16:41.054 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl enable ceph.target 2024-02-10T09:16:41.061 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T09:16:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl start ceph.target 2024-02-10T09:16:41.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl enable ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target 2024-02-10T09:16:41.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target. 2024-02-10T09:16:41.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target. 2024-02-10T09:16:41.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl start ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target 2024-02-10T09:16:41.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl daemon-reload 2024-02-10T09:16:41.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a 2024-02-10T09:16:41.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl reset-failed ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a 2024-02-10T09:16:41.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Failed to reset failed state of unit ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T09:16:41.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl enable ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a 2024-02-10T09:16:41.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Created symlink /etc/systemd/system/ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target.wants/ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-09b43244-c7f5-11ee-95b8-87774f69a715@.service. 2024-02-10T09:16:41.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl start ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a 2024-02-10T09:16:41.486 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:41 smithi153 systemd[1]: Starting Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:16:41.486 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:41 smithi153 podman[35378]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T09:16:41.486 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:41 smithi153 systemd[1]: Started Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:16:41.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-enabled firewalld.service 2024-02-10T09:16:41.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout disabled 2024-02-10T09:16:41.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-active firewalld.service 2024-02-10T09:16:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout inactive 2024-02-10T09:16:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:firewalld.service is not enabled 2024-02-10T09:16:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for mon to start... 2024-02-10T09:16:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for mon... 2024-02-10T09:16:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 status 2024-02-10T09:16:42.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout cluster: 2024-02-10T09:16:42.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout id: 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout health: HEALTH_OK 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout services: 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon: 1 daemons, quorum a (age 0.155761s) 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mgr: no daemons active 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout osd: 0 osds: 0 up, 0 in 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout data: 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout pools: 0 pools, 0 pgs 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout objects: 0 objects, 0 B 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout pgs: 2024-02-10T09:16:42.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:42.603 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:42 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.107689+0000 mon.a (mon.0) 0 : [INF] mkfs 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:42.603 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:42 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.082311+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T09:16:42.881 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Assimilating anything we can from ceph.conf... 2024-02-10T09:16:42.881 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '-v', '/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'assimilate-conf', '-i', '/var/lib/ceph/mon/ceph-a/config'] 2024-02-10T09:16:42.881 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z -v /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2024-02-10T09:16:43.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:43 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.106279+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T09:16:43.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:43 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.106535+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} 2024-02-10T09:16:43.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:43 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.110044+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-10T09:16:43.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:43 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.113700+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T09:16:43.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:43 smithi153 bash[35396]: cluster 2024-02-10T09:16:42.114168+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T09:16:43.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:43 smithi153 bash[35396]: audit 2024-02-10T09:16:42.262007+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.153:0/2483271833' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:16:43.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:43.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout [global] 2024-02-10T09:16:43.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout fsid = 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout log_to_journald = false 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_election_default_strategy = 1 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_host = [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_osd_allow_pg_remap = true 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_osd_allow_primary_affinity = true 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_warn_on_no_sortbitwise = false 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout osd_crush_chooseleaf_type = 0 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout [mgr] 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mgr/telemetry/nag = false 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout [mon] 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_warn_on_insecure_global_id_reclaim = false 2024-02-10T09:16:43.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-10T09:16:43.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:43.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout [osd] 2024-02-10T09:16:43.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout osd_map_max_advance = 10 2024-02-10T09:16:43.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout osd_memory_target_autotune = true 2024-02-10T09:16:43.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout osd_sloppy_crc = true 2024-02-10T09:16:44.037 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Generating new minimal ceph.conf... 2024-02-10T09:16:44.037 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '-v', '/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'generate-minimal-conf', '-o', '/var/lib/ceph/mon/ceph-a/config'] 2024-02-10T09:16:44.037 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z -v /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a:/var/lib/ceph/mon/ceph-a:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2024-02-10T09:16:44.438 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:44 smithi153 bash[35396]: audit 2024-02-10T09:16:43.437841+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.153:0/1943952182' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T09:16:44.438 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:44 smithi153 bash[35396]: audit 2024-02-10T09:16:43.440428+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.153:0/1943952182' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T09:16:45.036 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Restarting the monitor... 2024-02-10T09:16:45.037 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a 2024-02-10T09:16:45.334 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: Stopping Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:16:45.334 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 bash[35396]: audit 2024-02-10T09:16:44.580291+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.153:0/1674800332' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:16:45.334 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 bash[35396]: debug 2024-02-10T09:16:45.147+0000 7fbef47d7700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:16:45.334 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 bash[35396]: debug 2024-02-10T09:16:45.147+0000 7fbef47d7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T09:16:45.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 podman[35723]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.a 2024-02-10T09:16:45.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Succeeded. 2024-02-10T09:16:45.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: Stopped Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:16:45.913 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Found left-over process 35423 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:16:45.913 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:16:45.913 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Found left-over process 35746 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:16:45.913 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:16:45.913 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: Starting Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:16:45.930 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Setting mon public_network... 2024-02-10T09:16:45.930 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'set', 'mon', 'public_network', '172.21.0.0/20'] 2024-02-10T09:16:45.930 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 config set mon public_network 172.21.0.0/20 2024-02-10T09:16:46.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 podman[35764]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T09:16:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Found left-over process 35423 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:16:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:16:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Found left-over process 35746 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:16:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:16:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:45 smithi153 systemd[1]: Started Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.526+0000 7f480044f6c0 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.526+0000 7f480044f6c0 0 ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc), process ceph-mon, pid 1 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.526+0000 7f480044f6c0 0 pidfile_write: ignore empty --pid-file 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 load: jerasure load: lrc load: isa 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 set rocksdb option compression = kNoCompression 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 set rocksdb option write_buffer_size = 33554432 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 set rocksdb option compression = kNoCompression 2024-02-10T09:16:46.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 0 set rocksdb option write_buffer_size = 33554432 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 1 rocksdb: do_open column families: [default] 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: RocksDB version: 6.1.2 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Compile date Mar 23 2020 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: DB SUMMARY 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: CURRENT file: CURRENT 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 109 Bytes 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 69445 ; 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:16:46.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.create_if_missing: 0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.env: 0x55fb3ba721a0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.info_log: 0x55fb3d4dc9c0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:16:46.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.db_log_dir: 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:16:46.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.write_buffer_manager: 0x55fb3e074a80 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:16:46.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.row_cache: None 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.wal_filter: None 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:16:46.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Compression algorithms supported: 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kZSTD supported: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.531+0000 7f480044f6c0 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:16:46.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]: 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.merge_operator: 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fb3d3f8130) 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cache_index_and_filter_blocks: 1 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cache_index_and_filter_blocks_with_high_priority: 1 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: pin_top_level_index_and_filter: 1 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: index_type: 0 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: data_block_index_type: 0 2024-02-10T09:16:46.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: hash_index_allow_collision: 1 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: checksum: 1 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: no_block_cache: 0 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_cache: 0x55fb3d432610 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_cache_name: BinnedLRUCache 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_cache_options: 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: capacity : 536870912 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: num_shard_bits : 4 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: strict_capacity_limit : 0 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: high_pri_pool_ratio: 0.000 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_cache_compressed: (nil) 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: persistent_cache: (nil) 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_size: 4096 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_size_deviation: 10 2024-02-10T09:16:46.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_restart_interval: 16 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: index_block_restart_interval: 1 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: metadata_block_size: 4096 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: partition_filters: 0 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: use_delta_encoding: 1 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: whole_key_filtering: 1 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: verify_compression: 0 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: read_amp_bytes_per_bit: 0 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: format_version: 2 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: enable_index_compression: 1 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: block_align: 0 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:16:46.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.num_levels: 7 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:16:46.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:16:46.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:16:46.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:16:46.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: Options.ttl: 0 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 4 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707556606533519, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.532+0000 7f480044f6c0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #6 mode 2 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.533+0000 7f480044f6c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707556606534291, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 66486, "table_properties": {"data_size": 64781, "index_size": 310, "filter_size": 581, "raw_key_size": 8761, "raw_average_key_size": 46, "raw_value_size": 59740, "raw_average_value_size": 319, "num_data_blocks": 10, "num_entries": 187, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.533+0000 7f480044f6c0 4 rocksdb: [db/version_set.cc:3036] Creating manifest 8 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.533+0000 7f480044f6c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707556606534625, "job": 1, "event": "recovery_finished"} 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.534+0000 7f480044f6c0 4 rocksdb: DB pointer 0x55fb3d4cd200 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.534+0000 7f47e759c700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.534+0000 7f47e759c700 4 rocksdb: [db/db_impl.cc:778] 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** DB Stats ** 2024-02-10T09:16:46.955 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** Compaction Stats [default] ** 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: L0 2/0 66.55 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Sum 2/0 66.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** Compaction Stats [default] ** 2024-02-10T09:16:46.956 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Cumulative compaction: 0.00 GB write, 22.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Interval compaction: 0.00 GB write, 22.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:16:46.957 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** Compaction Stats [default] ** 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: L0 2/0 66.55 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Sum 2/0 66.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** Compaction Stats [default] ** 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:46.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Cumulative compaction: 0.00 GB write, 22.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.535+0000 7f480044f6c0 0 starting mon.a rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.536+0000 7f480044f6c0 1 mon.a@-1(???) e1 preinit fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.536+0000 7f480044f6c0 0 mon.a@-1(???).mds e1 new map 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.536+0000 7f480044f6c0 0 mon.a@-1(???).mds e1 print_map 2024-02-10T09:16:46.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: e1 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: enable_multiple, ever_enabled_multiple: 0,0 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: legacy client fscid: -1 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: No filesystems configured 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: audit 2024-02-10T09:16:44.580291+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.153:0/1674800332' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expand map: {default=false} 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta from 'false' to 'false' 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expanded map: {default=false} 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expand map: {default=info} 2024-02-10T09:16:46.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta from 'info' to 'info' 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expanded map: {default=info} 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expand map: {default=daemon} 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expanded map: {default=daemon} 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expand map: {default=debug} 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta from 'debug' to 'debug' 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: debug 2024-02-10T09:16:46.537+0000 7f480044f6c0 20 expand_channel_meta expanded map: {default=debug} 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cluster 2024-02-10T09:16:46.553334+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cluster 2024-02-10T09:16:46.553375+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cluster 2024-02-10T09:16:46.553405+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cluster 2024-02-10T09:16:46.553423+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T09:16:46.961 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:46 smithi153 bash[35780]: cluster 2024-02-10T09:16:46.553735+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T09:16:47.365 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Creating mgr... 2024-02-10T09:16:47.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl daemon-reload 2024-02-10T09:16:47.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T09:16:47.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl reset-failed ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T09:16:47.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Failed to reset failed state of unit ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T09:16:47.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl enable ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T09:16:47.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stderr Created symlink /etc/systemd/system/ceph-09b43244-c7f5-11ee-95b8-87774f69a715.target.wants/ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-09b43244-c7f5-11ee-95b8-87774f69a715@.service. 2024-02-10T09:16:47.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl start ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T09:16:47.619 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:47 smithi153 bash[35780]: audit 2024-02-10T09:16:46.796945+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.153:0/955836070' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "public_network", "value": "172.21.0.0/20"}]: dispatch 2024-02-10T09:16:47.619 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:47 smithi153 bash[35780]: audit 2024-02-10T09:16:46.799764+0000 mon.a (mon.0) 7 : audit [INF] from='client.? 172.21.15.153:0/955836070' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "public_network", "value": "172.21.0.0/20"}]': finished 2024-02-10T09:16:47.619 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:16:47 smithi153 systemd[1]: Starting Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:16:47.619 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:16:47 smithi153 podman[35997]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.y" found: no such container 2024-02-10T09:16:47.620 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:16:47 smithi153 systemd[1]: Started Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:16:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-enabled firewalld.service 2024-02-10T09:16:47.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout disabled 2024-02-10T09:16:47.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: systemctl is-active firewalld.service 2024-02-10T09:16:47.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:systemctl:stdout inactive 2024-02-10T09:16:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:firewalld.service is not enabled 2024-02-10T09:16:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:16:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Wrote config to /etc/ceph/ceph.conf 2024-02-10T09:16:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for mgr to start... 2024-02-10T09:16:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for mgr... 2024-02-10T09:16:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'status', '-f', 'json-pretty'] 2024-02-10T09:16:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 status -f json-pretty 2024-02-10T09:16:48.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "fsid": "09b43244-c7f5-11ee-95b8-87774f69a715", 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "health": { 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status": "HEALTH_OK", 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "checks": {}, 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mutes": [] 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "election_epoch": 5, 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum": [ 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 0 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum_names": [ 2024-02-10T09:16:48.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "a" 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum_age": 2, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "monmap": { 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_mon_release_name": "octopus", 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_mons": 1 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osdmap": { 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_osds": 0, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_up_osds": 0, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_up_since": 0, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_in_osds": 0, 2024-02-10T09:16:48.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_in_since": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_remapped_pgs": 0 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pgmap": { 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pgs_by_state": [], 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_pgs": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_pools": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_objects": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "data_bytes": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_used": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_avail": 0, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_total": 0 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "fsmap": { 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:48.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "by_rank": [], 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "up:standby": 0 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mgrmap": { 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available": false, 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_standbys": 0, 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modules": [ 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "iostat", 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "restful" 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {} 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "servicemap": { 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modified": "2024-02-10T09:16:42.108162+0000", 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {} 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:48.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress_events": {} 2024-02-10T09:16:48.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:48.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:48 smithi153 bash[35780]: audit 2024-02-10T09:16:48.625060+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.153:0/3693106994' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T09:16:49.284 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:mgr not available, waiting (1/10)... 2024-02-10T09:16:50.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'status', '-f', 'json-pretty'] 2024-02-10T09:16:50.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 status -f json-pretty 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "fsid": "09b43244-c7f5-11ee-95b8-87774f69a715", 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "health": { 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status": "HEALTH_OK", 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "checks": {}, 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mutes": [] 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.874 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "election_epoch": 5, 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum": [ 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 0 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum_names": [ 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "a" 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum_age": 4, 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "monmap": { 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_mon_release_name": "octopus", 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_mons": 1 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osdmap": { 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_osds": 0, 2024-02-10T09:16:50.875 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_up_osds": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_up_since": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_in_osds": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_in_since": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_remapped_pgs": 0 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pgmap": { 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pgs_by_state": [], 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_pgs": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_pools": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_objects": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "data_bytes": 0, 2024-02-10T09:16:50.876 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_used": 0, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_avail": 0, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_total": 0 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "fsmap": { 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "by_rank": [], 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "up:standby": 0 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mgrmap": { 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available": false, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_standbys": 0, 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modules": [ 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "iostat", 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "restful" 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {} 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "servicemap": { 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modified": "2024-02-10T09:16:42.108162+0000", 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {} 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress_events": {} 2024-02-10T09:16:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:51.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:50 smithi153 bash[35780]: audit 2024-02-10T09:16:50.867997+0000 mon.a (mon.0) 9 : audit [DBG] from='client.? 172.21.15.153:0/3494169660' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T09:16:51.506 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:mgr not available, waiting (2/10)... 2024-02-10T09:16:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'status', '-f', 'json-pretty'] 2024-02-10T09:16:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 status -f json-pretty 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: cluster 2024-02-10T09:16:51.649499+0000 mon.a (mon.0) 10 : cluster [INF] Activating manager daemon y 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: cluster 2024-02-10T09:16:51.654706+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00530804s) 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.655336+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.655494+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.655640+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.655867+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.656078+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: cluster 2024-02-10T09:16:51.669986+0000 mon.a (mon.0) 17 : cluster [INF] Manager daemon y is now available 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.682179+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.909331+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: cluster 2024-02-10T09:16:51.909371+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-10T09:16:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.909475+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/telemetry/report_id","val":"c84980b7-ef34-4cc7-98aa-b58305e1044a"}]: dispatch 2024-02-10T09:16:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.910741+0000 mon.a (mon.0) 22 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:16:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.914125+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/telemetry/report_id","val":"c84980b7-ef34-4cc7-98aa-b58305e1044a"}]': finished 2024-02-10T09:16:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.914794+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/telemetry/salt","val":"68847b8a-9c37-4394-8c12-0c327bb4b49a"}]: dispatch 2024-02-10T09:16:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:52 smithi153 bash[35780]: audit 2024-02-10T09:16:51.919500+0000 mon.a (mon.0) 25 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/telemetry/salt","val":"68847b8a-9c37-4394-8c12-0c327bb4b49a"}]': finished 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "fsid": "09b43244-c7f5-11ee-95b8-87774f69a715", 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "health": { 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status": "HEALTH_OK", 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "checks": {}, 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mutes": [] 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "election_epoch": 5, 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum": [ 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 0 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:53.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum_names": [ 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "a" 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "quorum_age": 7, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "monmap": { 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_mon_release_name": "octopus", 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_mons": 1 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osdmap": { 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 3, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_osds": 0, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_up_osds": 0, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_up_since": 0, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_in_osds": 0, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_in_since": 0, 2024-02-10T09:16:53.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_remapped_pgs": 0 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pgmap": { 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pgs_by_state": [], 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_pgs": 0, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_pools": 0, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_objects": 0, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "data_bytes": 0, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_used": 0, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_avail": 0, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "bytes_total": 0 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "fsmap": { 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "by_rank": [], 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "up:standby": 0 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mgrmap": { 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available": true, 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_standbys": 0, 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modules": [ 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "iostat", 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "restful" 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {} 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "servicemap": { 2024-02-10T09:16:53.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 1, 2024-02-10T09:16:53.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modified": "2024-02-10T09:16:42.108162+0000", 2024-02-10T09:16:53.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {} 2024-02-10T09:16:53.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:53.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress_events": {} 2024-02-10T09:16:53.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:53.918 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:53 smithi153 bash[35780]: cluster 2024-02-10T09:16:52.660129+0000 mon.a (mon.0) 26 : cluster [DBG] mgrmap e3: y(active, since 1.01073s) 2024-02-10T09:16:53.918 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:53 smithi153 bash[35780]: audit 2024-02-10T09:16:52.912470+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 172.21.15.153:0/2735262727' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:16:53.919 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:53 smithi153 bash[35780]: cluster 2024-02-10T09:16:52.912517+0000 mon.a (mon.0) 28 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-10T09:16:53.919 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:53 smithi153 bash[35780]: audit 2024-02-10T09:16:52.914107+0000 mon.a (mon.0) 29 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T09:16:53.919 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:53 smithi153 bash[35780]: audit 2024-02-10T09:16:53.391784+0000 mon.a (mon.0) 30 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T09:16:53.919 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:53 smithi153 bash[35780]: audit 2024-02-10T09:16:53.586677+0000 mon.a (mon.0) 31 : audit [DBG] from='client.? 172.21.15.153:0/1497830028' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T09:16:54.179 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Enabling cephadm module... 2024-02-10T09:16:54.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'mgr', 'module', 'enable', 'cephadm'] 2024-02-10T09:16:54.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 mgr module enable cephadm 2024-02-10T09:16:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:55 smithi153 bash[35780]: cluster 2024-02-10T09:16:54.660791+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e4: y(active, since 3s) 2024-02-10T09:16:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:55 smithi153 bash[35780]: audit 2024-02-10T09:16:54.846028+0000 mon.a (mon.0) 33 : audit [INF] from='client.? 172.21.15.153:0/1094574995' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T09:16:55.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:16:55 smithi153 bash[36014]: ignoring --setuser ceph since I am not root 2024-02-10T09:16:55.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:16:55 smithi153 bash[36014]: ignoring --setgroup ceph since I am not root 2024-02-10T09:16:56.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'mgr', 'dump'] 2024-02-10T09:16:56.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 mgr dump 2024-02-10T09:16:56.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:56 smithi153 bash[35780]: audit 2024-02-10T09:16:55.661967+0000 mon.a (mon.0) 34 : audit [INF] from='client.? 172.21.15.153:0/1094574995' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T09:16:56.960 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:56 smithi153 bash[35780]: cluster 2024-02-10T09:16:55.662017+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e5: y(active, since 4s) 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 5, 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_gid": 14100, 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_name": "y", 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_addrs": { 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:6800", 2024-02-10T09:16:57.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1824985332 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v1", 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:6801", 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1824985332 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_addr": "172.21.15.153:6801/1824985332", 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_change": "2024-02-10T09:16:51.649387+0000", 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_mgr_features": 4540138292836696063, 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available": true, 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "standbys": [], 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modules": [ 2024-02-10T09:16:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "cephadm", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "iostat", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "restful" 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available_modules": [ 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "alerts", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "How frequently to reexamine health status", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.357 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_destination": { 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_destination", 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Email address to send alerts to", 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_from_name": { 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_from_name", 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "Ceph", 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Email From: name", 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_host": { 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_host", 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "SMTP server", 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_password": { 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_password", 2024-02-10T09:16:57.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Password to authenticate with", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_port": { 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_port", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "465", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "SMTP port", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_sender": { 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_sender", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "SMTP envelope sender", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_ssl": { 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_ssl", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Use SSL to connect to SMTP server", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_user": { 2024-02-10T09:16:57.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_user", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "User to authenticate as", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "balancer", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active": { 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "active", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "automatically balance PGs across cluster", 2024-02-10T09:16:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "begin_time": { 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "begin_time", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0000", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "beginning time of day to automatically balance", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "This is a time of day in the format HHMM.", 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "begin_weekday": { 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "begin_weekday", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "0", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "7", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Restrict automatic balancing to this day of the week or later", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush_compat_max_iterations": { 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crush_compat_max_iterations", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "25", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "1", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "250", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "maximum number of iterations to attempt optimization", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush_compat_metrics": { 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crush_compat_metrics", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "pgs,objects,bytes", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "metrics with which to calculate OSD utilization", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush_compat_step": { 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crush_compat_step", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "float", 2024-02-10T09:16:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0.5", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "0.001", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "0.999", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "aggressiveness of optimization", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "end_time": { 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "end_time", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "2400", 2024-02-10T09:16:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "ending time of day to automatically balance", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "This is a time of day in the format HHMM.", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "end_weekday": { 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "end_weekday", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "7", 2024-02-10T09:16:57.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "0", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "7", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_score": { 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "min_score", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "float", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "minimum score, below which no optimization is attempted", 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mode": { 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mode", 2024-02-10T09:16:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "none", 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush-compat", 2024-02-10T09:16:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "none", 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "upmap" 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Balancer mode", 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pool_ids": { 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pool_ids", 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "pools which the automatic balancing will be limited to", 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:16:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to wake up and attempt optimization", 2024-02-10T09:16:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "upmap_max_deviation": { 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "upmap_max_deviation", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "1", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "deviation below which no optimization is attempted", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.389 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "upmap_max_optimizations": { 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "upmap_max_optimizations", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "maximum upmap optimizations to make per attempt", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.390 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "cephadm", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "allow_ptrace": { 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "allow_ptrace", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "allow SYS_PTRACE capability on ceph containers", 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "container_image_base": { 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "container_image_base", 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "ceph/ceph", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Container image name, without the tag", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "daemon_cache_timeout": { 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "daemon_cache_timeout", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "seconds to cache service (daemon) inventory", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "device_cache_timeout": { 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "device_cache_timeout", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "1800", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "seconds to cache device inventory", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "host_check_interval": { 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "host_check_interval", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to perform a host check", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "log to the \"cephadm\" cluster log channel\"", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.398 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.399 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mode": { 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mode", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "root", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "cephadm-package", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "root" 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "mode for remote execution of cephadm", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssh_config_file": { 2024-02-10T09:16:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssh_config_file", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "customized SSH config file to connect to managed hosts", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_on_failed_host_check": { 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_on_failed_host_check", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise a health warning if the host check fails", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_on_stray_daemons": { 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_on_stray_daemons", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_on_stray_hosts": { 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_on_stray_hosts", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crash", 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "retain_interval": { 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "retain_interval", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "31536000", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how long to retain crashes before pruning them", 2024-02-10T09:16:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_recent_interval": { 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_recent_interval", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "1209600", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "time interval in which to warn about recent crashes", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "dashboard", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ALERTMANAGER_API_HOST": { 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ALERTMANAGER_API_HOST", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.415 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "AUDIT_API_ENABLED": { 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "AUDIT_API_ENABLED", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "AUDIT_API_LOG_PAYLOAD": { 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "AUDIT_API_LOG_PAYLOAD", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ENABLE_BROWSABLE_API": { 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ENABLE_BROWSABLE_API", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.418 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_cephfs": { 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_cephfs", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.419 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_iscsi": { 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_iscsi", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.420 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_mirroring": { 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_mirroring", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_nfs": { 2024-02-10T09:16:57.421 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_nfs", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_rbd": { 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_rbd", 2024-02-10T09:16:57.422 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_rgw": { 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_rgw", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.423 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.424 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_PASSWORD": { 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_PASSWORD", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "admin", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_SSL_VERIFY": { 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_SSL_VERIFY", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_URL": { 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_URL", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.427 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_USERNAME": { 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_USERNAME", 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "admin", 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_UPDATE_DASHBOARDS": { 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.430 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ISCSI_API_SSL_VERIFICATION": { 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ISCSI_API_SSL_VERIFICATION", 2024-02-10T09:16:57.431 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.432 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PROMETHEUS_API_HOST": { 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PROMETHEUS_API_HOST", 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.433 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.434 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.436 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.437 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.440 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_ENABLED": { 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_ENABLED", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.441 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_EXCLUSION_LIST": { 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_EXCLUSION_LIST", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_MIN_COMPLEXITY": { 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-02-10T09:16:57.442 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_MIN_LENGTH": { 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_MIN_LENGTH", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.443 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "REST_REQUESTS_TIMEOUT": { 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "REST_REQUESTS_TIMEOUT", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.444 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "45", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_ACCESS_KEY": { 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_ACCESS_KEY", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.445 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_ADMIN_RESOURCE": { 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_ADMIN_RESOURCE", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "admin", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.446 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_HOST": { 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_HOST", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.447 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_PORT": { 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_PORT", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "80", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.448 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_SCHEME": { 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_SCHEME", 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.449 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "http", 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.450 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_SECRET_KEY": { 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_SECRET_KEY", 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.451 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_SSL_VERIFY": { 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_SSL_VERIFY", 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.452 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.453 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_USER_ID": { 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_USER_ID", 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.454 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "USER_PWD_EXPIRATION_SPAN": { 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "USER_PWD_EXPIRATION_SPAN", 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.455 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "USER_PWD_EXPIRATION_WARNING_1": { 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:16:57.456 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "USER_PWD_EXPIRATION_WARNING_2": { 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crt_file": { 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crt_file", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.458 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug": { 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "debug", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Enable/disable debug options", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.459 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "jwt_token_ttl": { 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "jwt_token_ttl", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "28800", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.460 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "key_file": { 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "key_file", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.461 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.462 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.463 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.464 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.465 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "password": { 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "password", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.466 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_addr": { 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_addr", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "::", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.467 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_port": { 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_port", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8080", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.468 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssl": { 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssl", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssl_server_port": { 2024-02-10T09:16:57.469 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssl_server_port", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8443", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "standby_behaviour": { 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "standby_behaviour", 2024-02-10T09:16:57.470 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "redirect", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "redirect" 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "standby_error_status_code": { 2024-02-10T09:16:57.471 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "standby_error_status_code", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "500", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "400", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "599", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "url_prefix": { 2024-02-10T09:16:57.472 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "url_prefix", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "username": { 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "username", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.473 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "devicehealth", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.474 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enable_monitoring": { 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "enable_monitoring", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "monitor device health metrics", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.475 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.476 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.477 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.478 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.479 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mark_out_threshold": { 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mark_out_threshold", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "2419200", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.480 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "automatically mark OSD if it may fail before this long", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pool_name": { 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pool_name", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "device_health_metrics", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "name of pool in which to store device health metrics", 2024-02-10T09:16:57.481 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "retention_period": { 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "retention_period", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "15552000", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how long to retain device health metrics", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.482 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "scrape_frequency": { 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "scrape_frequency", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "86400", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to scrape device health metrics", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.483 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "self_heal": { 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "self_heal", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "preemptively heal cluster around devices that may fail", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:16:57.484 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to wake up and check device health", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.485 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_threshold": { 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_threshold", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "7257600", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise health warning if OSD may fail before this long", 2024-02-10T09:16:57.487 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "diskprediction_local", 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.488 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.489 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.490 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.491 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.492 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "predict_interval": { 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "predict_interval", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "86400", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.493 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "predictor_model": { 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "predictor_model", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "prophetstor", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.494 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:16:57.495 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "influx", 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": false, 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "influxdb python module not found", 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.496 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "batch_size": { 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "batch_size", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5000", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "database": { 2024-02-10T09:16:57.497 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "database", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "ceph", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "hostname": { 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "hostname", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.498 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.499 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "30", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.500 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.501 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.502 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "password": { 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "password", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.505 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "port": { 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "port", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8086", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssl": { 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssl", 2024-02-10T09:16:57.506 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "false", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "threads": { 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "threads", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.507 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "username": { 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "username", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.508 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "verify_ssl": { 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "verify_ssl", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "true", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.509 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "insights", 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.510 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.511 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.512 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.513 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.514 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "iostat", 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.515 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.517 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.518 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "k8sevents", 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ceph_event_retention_days": { 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ceph_event_retention_days", 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.520 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "7", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Days to hold ceph event information within local cache", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "config_check_secs": { 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "config_check_secs", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:16:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "10", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "interval (secs) to check for cluster configuration changes", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.526 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "localpool", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "failure_domain": { 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "failure_domain", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "host", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "failure domain for any created local pool", 2024-02-10T09:16:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.531 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_size": { 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "min_size", 2024-02-10T09:16:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "default min_size for any created local pool", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_rep": { 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "num_rep", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "3", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "default replica count for any created local pool", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pg_num": { 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pg_num", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.534 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "128", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "default pg_num for any created local pool", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "prefix": { 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "prefix", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.535 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "name prefix for any created local pool", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "subtree": { 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "subtree", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.536 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "rack", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "CRUSH level for which to create a local pool", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "orchestrator", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.537 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "orchestrator": { 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "orchestrator", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "cephadm", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rook", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "test_orchestrator" 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Orchestrator backend", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "osd_support", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pg_autoscaler", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "progress", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.558 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.559 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max_completed_events": { 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "max_completed_events", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "50", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "number of past completed events to remember", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "persist_interval": { 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "persist_interval", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to persist completed events", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.561 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "prometheus", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.562 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.563 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.565 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.566 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_stats_pools": { 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rbd_stats_pools", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.567 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_stats_pools_refresh_interval": { 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rbd_stats_pools_refresh_interval", 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.568 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "scrape_interval": { 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "scrape_interval", 2024-02-10T09:16:57.569 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.570 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_addr": { 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_addr", 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.571 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_port": { 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_port", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rbd_support", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.574 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.575 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.576 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.577 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mirror_snapshot_schedule": { 2024-02-10T09:16:57.578 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mirror_snapshot_schedule", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "trash_purge_schedule": { 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "trash_purge_schedule", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.579 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "restful", 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.580 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "key_file": { 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "key_file", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.581 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.582 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.583 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.584 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.585 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_addr": { 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_addr", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_port": { 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_port", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.587 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rook", 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.588 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.589 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.592 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "selftest", 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.594 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.595 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.596 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "roption1": { 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "roption1", 2024-02-10T09:16:57.598 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "roption2": { 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "roption2", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.599 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "xyz", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption1": { 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption1", 2024-02-10T09:16:57.600 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption2": { 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption2", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.601 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption3": { 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption3", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "float", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.602 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption4": { 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption4", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.603 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption5": { 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption5", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.604 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption6": { 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption6", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.605 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "testkey": { 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "testkey", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "testlkey": { 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "testlkey", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.607 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "testnewline": { 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "testnewline", 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.608 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "status", 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.610 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.611 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.612 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.613 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.614 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.615 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "telegraf", 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "address": { 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "address", 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.616 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "unixgram:///tmp/telegraf.sock", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.617 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "15", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.618 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.619 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.620 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.621 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.622 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "telemetry", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_basic": { 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_basic", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.623 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Share basic cluster information (size, version)", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_crash": { 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_crash", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.624 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_device": { 2024-02-10T09:16:57.625 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_device", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.626 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_ident": { 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_ident", 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "contact": { 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "contact", 2024-02-10T09:16:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.629 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "description": { 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "description", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "device_url": { 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "device_url", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "https://telemetry.ceph.com/device", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enabled": { 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "enabled", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.632 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "24", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "8", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "last_opt_revision": { 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "last_opt_revision", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "1", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "leaderboard": { 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "leaderboard", 2024-02-10T09:16:57.634 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.635 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.636 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.639 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "organization": { 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "organization", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "proxy": { 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "proxy", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "url": { 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "url", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "https://telemetry.ceph.com/report", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "test_orchestrator", 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.645 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.646 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.647 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.648 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "volumes", 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.649 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.650 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.651 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.652 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.653 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.654 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix", 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "discovery_interval": { 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "discovery_interval", 2024-02-10T09:16:57.655 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "100", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "identifier": { 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "identifier", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.656 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.657 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:16:57.658 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:16:57.659 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:16:57.660 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.661 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "zabbix_host": { 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix_host", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "zabbix_port": { 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix_port", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10051", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "zabbix_sender": { 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix_sender", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "/usr/bin/zabbix_sender", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {}, 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "always_on_modules": { 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nautilus": [ 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "balancer", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crash", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "devicehealth", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "orchestrator_cli", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_support", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "volumes" 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "octopus": [ 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "balancer", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crash", 2024-02-10T09:16:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "devicehealth", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "orchestrator", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_support", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pg_autoscaler", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_support", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "telemetry", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "volumes" 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "last_failure_osd_epoch": 0, 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_clients": [ 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:16:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:0", 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 782778237 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:0", 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1500108103 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:16:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:0", 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1431131370 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:16:57.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:57 smithi153 bash[35780]: audit 2024-02-10T09:16:56.953515+0000 mon.a (mon.0) 36 : audit [DBG] from='client.? 172.21.15.153:0/2491710757' entity='client.admin' cmd=[{"prefix": "mgr dump"}]: dispatch 2024-02-10T09:16:59.637 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for the mgr to restart... 2024-02-10T09:16:59.637 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for Mgr epoch 5... 2024-02-10T09:16:59.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'tell', 'mgr', 'mgr_status'] 2024-02-10T09:16:59.638 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 tell mgr mgr_status 2024-02-10T09:16:59.639 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:59 smithi153 bash[35780]: cluster 2024-02-10T09:16:59.319405+0000 mon.a (mon.0) 37 : cluster [INF] Active manager daemon y restarted 2024-02-10T09:16:59.639 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:59 smithi153 bash[35780]: cluster 2024-02-10T09:16:59.319799+0000 mon.a (mon.0) 38 : cluster [INF] Activating manager daemon y 2024-02-10T09:16:59.639 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:16:59 smithi153 bash[35780]: cluster 2024-02-10T09:16:59.321549+0000 mon.a (mon.0) 39 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: cluster 2024-02-10T09:16:59.372020+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0523008s) 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.374140+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.374784+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.374936+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.375068+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.375330+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.377424+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.377667+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: cluster 2024-02-10T09:16:59.390966+0000 mon.a (mon.0) 48 : cluster [INF] Manager daemon y is now available 2024-02-10T09:17:00.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.410583+0000 mon.a (mon.0) 49 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T09:17:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:00 smithi153 bash[35780]: audit 2024-02-10T09:16:59.893948+0000 mon.a (mon.0) 50 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T09:17:01.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:01 smithi153 bash[35780]: cluster 2024-02-10T09:17:00.384391+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e7: y(active, since 1.06467s) 2024-02-10T09:17:02.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:02 smithi153 bash[35780]: cluster 2024-02-10T09:17:01.387438+0000 mon.a (mon.0) 52 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-02-10T09:17:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mgrmap_epoch": 8, 2024-02-10T09:17:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "initialized": true 2024-02-10T09:17:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:06.937 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Setting orchestrator backend to cephadm... 2024-02-10T09:17:06.937 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'orch', 'set', 'backend', 'cephadm'] 2024-02-10T09:17:06.937 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 orch set backend cephadm 2024-02-10T09:17:09.584 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:09 smithi153 bash[35780]: audit 2024-02-10T09:17:09.273252+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix":"config set","value":"cephadm","who":"mgr","name":"mgr/orchestrator/orchestrator"}]: dispatch 2024-02-10T09:17:09.584 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:09 smithi153 bash[35780]: audit 2024-02-10T09:17:09.275664+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd='[{"prefix":"config set","value":"cephadm","who":"mgr","name":"mgr/orchestrator/orchestrator"}]': finished 2024-02-10T09:17:09.869 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Generating ssh key... 2024-02-10T09:17:09.869 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'cephadm', 'generate-key'] 2024-02-10T09:17:09.869 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 cephadm generate-key 2024-02-10T09:17:10.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[35780]: audit 2024-02-10T09:17:09.272113+0000 mgr.y (mgr.14117) 1 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:11.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: Generating public/private rsa key pair. 2024-02-10T09:17:11.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: Your identification has been saved in /tmp/tmp__0dhf6i/key. 2024-02-10T09:17:11.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: Your public key has been saved in /tmp/tmp__0dhf6i/key.pub. 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: The key fingerprint is: 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: SHA256:erNSFHWHhjPSymnbsIQjH4uAtLGlmFttWCICEML3J7w ceph-09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: The key's randomart image is: 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: +---[RSA 3072]----+ 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: |O. o..... | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: |++.o. o =.o. | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: |++B=o o = + | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: |+++ +++.O | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: | o o ++BS= | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: |. .Eo.+ . | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: | ..o | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: | .. o | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: | .. | 2024-02-10T09:17:11.123 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:10 smithi153 bash[36014]: +----[SHA256]-----+ 2024-02-10T09:17:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'cephadm', 'get-pub-key'] 2024-02-10T09:17:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 cephadm get-pub-key 2024-02-10T09:17:11.429 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:11 smithi153 bash[35780]: audit 2024-02-10T09:17:10.533849+0000 mgr.y (mgr.14117) 2 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:11.429 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:11 smithi153 bash[35780]: cephadm 2024-02-10T09:17:10.534076+0000 mgr.y (mgr.14117) 3 : cephadm [INF] Generating ssh key... 2024-02-10T09:17:11.429 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:11 smithi153 bash[35780]: audit 2024-02-10T09:17:10.809434+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/ssh_identity_key","val":"-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAYEAxPTvpmfVwctk4zv9ig45yuceuEZVm14lnCfIAMJeFXbQt3hYOBKL\nx/oM6kFIWPpAjzC62bkyH1Kno40tFhYEktR6jEstCCfVDaSXd6te3e7goeC2bfIJgr76/B\nel0rZWr7Xe9nuPimW6tpMPhHnNyB3i/g2jdNpfmNF6ox7ONLCEIGMfc8MFQYolwFjpokSp\n4wuPdcDYI6LcH3cid15mdFc3VU7SdSmvs4mIb/gnsAHdBvCqYguE2cridpcNmIzT4fChmE\noq4K8MaOdZtLQSwudJCFv5Qb40a3JpcSCZqkgQ6CRJsNcIK8o4JnyKzzPbc06EM9F4OGpR\nLA5XcT4UhTwkv1Bd+FKwvrT133R560I5S00A7etV1ZsDPWzLUKPDwKaEwxkjnEVxpg+qVG\nQO1g4fh237lq4N5Z1p8QApDvjLQdXbxjG0pEK+MFVeyX0AYXjPMoiFI41lWlG1mz0tRqBw\n8eNoWxicWGlYfkDEMgf17yK34efNOBKY9euCtwyNAAAFoLqez1e6ns9XAAAAB3NzaC1yc2\nEAAAGBAMT076Zn1cHLZOM7/YoOOcrnHrhGVZteJZwnyADCXhV20Ld4WDgSi8f6DOpBSFj6\nQI8wutm5Mh9Sp6ONLRYWBJLUeoxLLQgn1Q2kl3erXt3u4KHgtm3yCYK++vwXpdK2Vq+13v\nZ7j4pluraTD4R5zcgd4v4No3TaX5jReqMezjSwhCBjH3PDBUGKJcBY6aJEqeMLj3XA2COi\n3B93IndeZnRXN1VO0nUpr7OJiG/4J7AB3QbwqmILhNnK4naXDZiM0+HwoZhKKuCvDGjnWb\nS0EsLnSQhb+UG+NGtyaXEgmapIEOgkSbDXCCvKOCZ8is8z23NOhDPReDhqUSwOV3E+FIU8\nJL9QXfhSsL609d90eetCOUtNAO3rVdWbAz1sy1Cjw8CmhMMZI5xFcaYPqlRkDtYOH4dt+5\nauDeWdafEAKQ74y0HV28YxtKRCvjBVXsl9AGF4zzKIhSONZVpRtZs9LUagcPHjaFsYnFhp\nWH5AxDIH9e8it+HnzTgSmPXrgrcMjQAAAAMBAAEAAAGABXj807Xc5aaeJgY7tENS7cVgQC\nSMKmYCS4V7dDHyZUxI3mA0q1IwmhzjkPrBZN+HPbNP7c7MirxENjItB4kTFXVb9lv8Y1af\n/+IL52JWn3oMudLBnupzwzNFTA3oKKOyYu19xYGHaoQLHRmW+8paom3aKc79NLkAoQFpE/\nE+L46cDT5T2GaZ1cld5StTSpP32Hwwv0fdeZgD38fyqDhyRldKDU/EX0cKP8XZiaruh2B1\n4MR9J35EcbzGcB8gXJXFHOVHKy/7WYkp9dWyTVlsqcLCyWmm7OhFfQ6210vF8rMBood0NW\n6+YtWX7yRhHToESCiDW8Tc9rIqbij4OvTju0BZOpegI37vsgWZSkrnNIaMQF4UFWNqTfYJ\nc9aNyQY52BpEDToyyvqVcILNJeNMBTLy+t2DZuc7VMgdOnN+UjDvIu6xyEhHgP723pLuDh\nveKCpo5dNi8pM6HQNow04ITPJIAQwDfDDLGwvxjmH1psVTVz6P+9JlBxbo3aBNinoBAAAA\nwDHv7846Sdw/YZBBthRLIXD6VROHf6ynCzwUdkKwSx8KtXzUxIy7rR9ziQYLWABeX5Q02Z\nEffKjJ7OksRAOAqI3aZMS7PF3YxeOrGh5S0K8UFO5m95r1R+jvMqolr+jlgzATbLBH0smz\n+smpwMtMHgURPp0Qp2YYuN7NOspA3meNLc+MfrMSn79kbGGIPmpCkJjlZGtKgXBNurE4Ew\nfGfUJ/SORF9hxO88Y6VqUCYweYZj3l14wEB2S9ED6Za6VACQAAAMEA8k+69CzkhEcPBeN/\nu/BUN0cDo7aK764wm9vyUmLnEKA+8RNLC9G+8OgRLkNc/Czy3P1ZAoO0MDaEQblOIre0EC\n3zErlL1puE53JR3lFPqBWHjBOROKHpktyWBHh+hsY+9YI4nCrtH5HntmfSjoLBw3qrXh3o\nwI7TtXDxqr08m2HvEwOfb2fsRNgCAffBhoNmfbyY4mL6njPgsy8slwAHtUb3f6T+vtWFof\nWF6LUhhGeLr5zbSXRVUpdjxU110uWBAAAAwQDQFUs7gAdOspGzeevRuKL3HJv3oiZniz/v\nDLWJfz6xs5UEjL5qH5mzyXE78JpEvVAH76w1EX5dhHoRE5xdsOBd/R24R2YKykN265OSVN\nx10aDjlrgu0fimOlymv355K2txS1uL+VDkQePBOsyR8Vcf74KG3wIMcCts17KJW7K3dF8V\nZk8KKYyWLZnNrGR2BGN6yAMbq2g5aSqlg9l4P3OYBAHnxJf5ScuGFdQ5rD7nIMA0SjkLZN\n50ClOe81w15Q0AAAApY2VwaC0wOWI0MzI0NC1jN2Y1LTExZWUtOTViOC04Nzc3NGY2OWE3\nMTUBAg==\n-----END OPENSSH PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:17:11.429 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:11 smithi153 bash[35780]: audit 2024-02-10T09:17:10.811245+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/ssh_identity_key","val":"-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAYEAxPTvpmfVwctk4zv9ig45yuceuEZVm14lnCfIAMJeFXbQt3hYOBKL\nx/oM6kFIWPpAjzC62bkyH1Kno40tFhYEktR6jEstCCfVDaSXd6te3e7goeC2bfIJgr76/B\nel0rZWr7Xe9nuPimW6tpMPhHnNyB3i/g2jdNpfmNF6ox7ONLCEIGMfc8MFQYolwFjpokSp\n4wuPdcDYI6LcH3cid15mdFc3VU7SdSmvs4mIb/gnsAHdBvCqYguE2cridpcNmIzT4fChmE\noq4K8MaOdZtLQSwudJCFv5Qb40a3JpcSCZqkgQ6CRJsNcIK8o4JnyKzzPbc06EM9F4OGpR\nLA5XcT4UhTwkv1Bd+FKwvrT133R560I5S00A7etV1ZsDPWzLUKPDwKaEwxkjnEVxpg+qVG\nQO1g4fh237lq4N5Z1p8QApDvjLQdXbxjG0pEK+MFVeyX0AYXjPMoiFI41lWlG1mz0tRqBw\n8eNoWxicWGlYfkDEMgf17yK34efNOBKY9euCtwyNAAAFoLqez1e6ns9XAAAAB3NzaC1yc2\nEAAAGBAMT076Zn1cHLZOM7/YoOOcrnHrhGVZteJZwnyADCXhV20Ld4WDgSi8f6DOpBSFj6\nQI8wutm5Mh9Sp6ONLRYWBJLUeoxLLQgn1Q2kl3erXt3u4KHgtm3yCYK++vwXpdK2Vq+13v\nZ7j4pluraTD4R5zcgd4v4No3TaX5jReqMezjSwhCBjH3PDBUGKJcBY6aJEqeMLj3XA2COi\n3B93IndeZnRXN1VO0nUpr7OJiG/4J7AB3QbwqmILhNnK4naXDZiM0+HwoZhKKuCvDGjnWb\nS0EsLnSQhb+UG+NGtyaXEgmapIEOgkSbDXCCvKOCZ8is8z23NOhDPReDhqUSwOV3E+FIU8\nJL9QXfhSsL609d90eetCOUtNAO3rVdWbAz1sy1Cjw8CmhMMZI5xFcaYPqlRkDtYOH4dt+5\nauDeWdafEAKQ74y0HV28YxtKRCvjBVXsl9AGF4zzKIhSONZVpRtZs9LUagcPHjaFsYnFhp\nWH5AxDIH9e8it+HnzTgSmPXrgrcMjQAAAAMBAAEAAAGABXj807Xc5aaeJgY7tENS7cVgQC\nSMKmYCS4V7dDHyZUxI3mA0q1IwmhzjkPrBZN+HPbNP7c7MirxENjItB4kTFXVb9lv8Y1af\n/+IL52JWn3oMudLBnupzwzNFTA3oKKOyYu19xYGHaoQLHRmW+8paom3aKc79NLkAoQFpE/\nE+L46cDT5T2GaZ1cld5StTSpP32Hwwv0fdeZgD38fyqDhyRldKDU/EX0cKP8XZiaruh2B1\n4MR9J35EcbzGcB8gXJXFHOVHKy/7WYkp9dWyTVlsqcLCyWmm7OhFfQ6210vF8rMBood0NW\n6+YtWX7yRhHToESCiDW8Tc9rIqbij4OvTju0BZOpegI37vsgWZSkrnNIaMQF4UFWNqTfYJ\nc9aNyQY52BpEDToyyvqVcILNJeNMBTLy+t2DZuc7VMgdOnN+UjDvIu6xyEhHgP723pLuDh\nveKCpo5dNi8pM6HQNow04ITPJIAQwDfDDLGwvxjmH1psVTVz6P+9JlBxbo3aBNinoBAAAA\nwDHv7846Sdw/YZBBthRLIXD6VROHf6ynCzwUdkKwSx8KtXzUxIy7rR9ziQYLWABeX5Q02Z\nEffKjJ7OksRAOAqI3aZMS7PF3YxeOrGh5S0K8UFO5m95r1R+jvMqolr+jlgzATbLBH0smz\n+smpwMtMHgURPp0Qp2YYuN7NOspA3meNLc+MfrMSn79kbGGIPmpCkJjlZGtKgXBNurE4Ew\nfGfUJ/SORF9hxO88Y6VqUCYweYZj3l14wEB2S9ED6Za6VACQAAAMEA8k+69CzkhEcPBeN/\nu/BUN0cDo7aK764wm9vyUmLnEKA+8RNLC9G+8OgRLkNc/Czy3P1ZAoO0MDaEQblOIre0EC\n3zErlL1puE53JR3lFPqBWHjBOROKHpktyWBHh+hsY+9YI4nCrtH5HntmfSjoLBw3qrXh3o\nwI7TtXDxqr08m2HvEwOfb2fsRNgCAffBhoNmfbyY4mL6njPgsy8slwAHtUb3f6T+vtWFof\nWF6LUhhGeLr5zbSXRVUpdjxU110uWBAAAAwQDQFUs7gAdOspGzeevRuKL3HJv3oiZniz/v\nDLWJfz6xs5UEjL5qH5mzyXE78JpEvVAH76w1EX5dhHoRE5xdsOBd/R24R2YKykN265OSVN\nx10aDjlrgu0fimOlymv355K2txS1uL+VDkQePBOsyR8Vcf74KG3wIMcCts17KJW7K3dF8V\nZk8KKYyWLZnNrGR2BGN6yAMbq2g5aSqlg9l4P3OYBAHnxJf5ScuGFdQ5rD7nIMA0SjkLZN\n50ClOe81w15Q0AAAApY2VwaC0wOWI0MzI0NC1jN2Y1LTExZWUtOTViOC04Nzc3NGY2OWE3\nMTUBAg==\n-----END OPENSSH PRIVATE KEY-----\n"}]': finished 2024-02-10T09:17:11.429 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:11 smithi153 bash[35780]: audit 2024-02-10T09:17:10.811604+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/ssh_identity_pub","val":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715\n"}]: dispatch 2024-02-10T09:17:11.429 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:11 smithi153 bash[35780]: audit 2024-02-10T09:17:10.812815+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/ssh_identity_pub","val":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715\n"}]': finished 2024-02-10T09:17:12.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:17:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Wrote public SSH key to to /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:17:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Adding key to root@localhost's authorized_keys... 2024-02-10T09:17:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Adding host smithi153... 2024-02-10T09:17:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'orch', 'host', 'add', 'smithi153'] 2024-02-10T09:17:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 orch host add smithi153 2024-02-10T09:17:12.902 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:12 smithi153 bash[35780]: audit 2024-02-10T09:17:12.113725+0000 mgr.y (mgr.14117) 4 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:13.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:13 smithi153 bash[36014]: Warning: Permanently added 'smithi153,172.21.15.153' (ECDSA) to the list of known hosts. 2024-02-10T09:17:14.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:13 smithi153 bash[35780]: audit 2024-02-10T09:17:13.309242+0000 mgr.y (mgr.14117) 5 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout Added host 'smithi153' 2024-02-10T09:17:14.981 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Enabling the dashboard module... 2024-02-10T09:17:14.981 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'mgr', 'module', 'enable', 'dashboard'] 2024-02-10T09:17:14.981 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 mgr module enable dashboard 2024-02-10T09:17:15.076 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:14 smithi153 bash[35780]: audit 2024-02-10T09:17:13.778577+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"smithi153\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:17:15.077 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:14 smithi153 bash[35780]: audit 2024-02-10T09:17:13.781047+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"smithi153\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:17:15.077 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:14 smithi153 bash[35780]: cephadm 2024-02-10T09:17:13.781357+0000 mgr.y (mgr.14117) 6 : cephadm [INF] Added host smithi153 2024-02-10T09:17:15.077 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:14 smithi153 bash[35780]: audit 2024-02-10T09:17:13.975349+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {}, \"devices\": [], \"daemon_config_deps\": {}, \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:17:15.077 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:14 smithi153 bash[35780]: audit 2024-02-10T09:17:13.977792+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {}, \"devices\": [], \"daemon_config_deps\": {}, \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:17:15.077 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:14 smithi153 bash[35780]: audit 2024-02-10T09:17:13.978203+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:17:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:15 smithi153 bash[35780]: audit 2024-02-10T09:17:15.319864+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:15.318299\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:15.319373\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}}, \"devices\": [], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:15.319451\", \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:17:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:15 smithi153 bash[35780]: audit 2024-02-10T09:17:15.321694+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:15.318299\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:15.319373\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}}, \"devices\": [], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:15.319451\", \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:17:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:15 smithi153 bash[35780]: audit 2024-02-10T09:17:15.322105+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.14117 172.21.15.153:0/1538574503' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:17:17.068 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:16 smithi153 bash[35780]: audit 2024-02-10T09:17:16.100044+0000 mon.a (mon.0) 67 : audit [INF] from='client.? 172.21.15.153:0/2088875119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T09:17:17.069 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:16 smithi153 bash[36014]: ignoring --setuser ceph since I am not root 2024-02-10T09:17:17.069 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:16 smithi153 bash[36014]: ignoring --setgroup ceph since I am not root 2024-02-10T09:17:17.815 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'mgr', 'dump'] 2024-02-10T09:17:17.815 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 mgr dump 2024-02-10T09:17:18.065 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:17 smithi153 bash[35780]: audit 2024-02-10T09:17:16.756444+0000 mon.a (mon.0) 68 : audit [INF] from='client.? 172.21.15.153:0/2088875119' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T09:17:18.065 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:17 smithi153 bash[35780]: cluster 2024-02-10T09:17:16.756496+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e9: y(active, since 17s) 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "epoch": 9, 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_gid": 14117, 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_name": "y", 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_addrs": { 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:17:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:6800", 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1546421638 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v1", 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:6801", 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1546421638 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_addr": "172.21.15.153:6801/1546421638", 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_change": "2024-02-10T09:16:59.319711+0000", 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_mgr_features": 4540138292836696063, 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available": true, 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "standbys": [], 2024-02-10T09:17:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "modules": [ 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "cephadm", 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "dashboard", 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "iostat", 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "restful" 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "available_modules": [ 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "alerts", 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:17:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "How frequently to reexamine health status", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_destination": { 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_destination", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Email address to send alerts to", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_from_name": { 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_from_name", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "Ceph", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Email From: name", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_host": { 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_host", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "SMTP server", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_password": { 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_password", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Password to authenticate with", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_port": { 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_port", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "465", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "SMTP port", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_sender": { 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_sender", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "SMTP envelope sender", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_ssl": { 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_ssl", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Use SSL to connect to SMTP server", 2024-02-10T09:17:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "smtp_user": { 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "smtp_user", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "User to authenticate as", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "balancer", 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active": { 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "active", 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.111 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "automatically balance PGs across cluster", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "begin_time": { 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "begin_time", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0000", 2024-02-10T09:17:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "beginning time of day to automatically balance", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "This is a time of day in the format HHMM.", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "begin_weekday": { 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "begin_weekday", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "0", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "7", 2024-02-10T09:17:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Restrict automatic balancing to this day of the week or later", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush_compat_max_iterations": { 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crush_compat_max_iterations", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "25", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "1", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "250", 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "maximum number of iterations to attempt optimization", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush_compat_metrics": { 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crush_compat_metrics", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "pgs,objects,bytes", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "metrics with which to calculate OSD utilization", 2024-02-10T09:17:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush_compat_step": { 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crush_compat_step", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "float", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0.5", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "0.001", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "0.999", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "aggressiveness of optimization", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.116 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "end_time": { 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "end_time", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "2400", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "ending time of day to automatically balance", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "This is a time of day in the format HHMM.", 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "end_weekday": { 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "end_weekday", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "7", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "0", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "7", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.119 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.122 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_score": { 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "min_score", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "float", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "minimum score, below which no optimization is attempted", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mode": { 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mode", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "none", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crush-compat", 2024-02-10T09:17:19.124 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "none", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "upmap" 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Balancer mode", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pool_ids": { 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pool_ids", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.125 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "pools which the automatic balancing will be limited to", 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.126 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to wake up and attempt optimization", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "upmap_max_deviation": { 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "upmap_max_deviation", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:17:19.127 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "1", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "deviation below which no optimization is attempted", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "upmap_max_optimizations": { 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "upmap_max_optimizations", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "uint", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:17:19.128 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "maximum upmap optimizations to make per attempt", 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "cephadm", 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "allow_ptrace": { 2024-02-10T09:17:19.129 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "allow_ptrace", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "allow SYS_PTRACE capability on ceph containers", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "container_image_base": { 2024-02-10T09:17:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "container_image_base", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "ceph/ceph", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Container image name, without the tag", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "daemon_cache_timeout": { 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "daemon_cache_timeout", 2024-02-10T09:17:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "seconds to cache service (daemon) inventory", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "device_cache_timeout": { 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "device_cache_timeout", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.132 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "1800", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "seconds to cache device inventory", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "host_check_interval": { 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "host_check_interval", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:17:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to perform a host check", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.134 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.135 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "log to the \"cephadm\" cluster log channel\"", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.136 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.137 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mode": { 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mode", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.138 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "root", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "cephadm-package", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "root" 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "mode for remote execution of cephadm", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssh_config_file": { 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssh_config_file", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.139 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "customized SSH config file to connect to managed hosts", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_on_failed_host_check": { 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_on_failed_host_check", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.140 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise a health warning if the host check fails", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_on_stray_daemons": { 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_on_stray_daemons", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.141 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_on_stray_hosts": { 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_on_stray_hosts", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.142 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crash", 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.143 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.144 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.145 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.146 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.147 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "retain_interval": { 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "retain_interval", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.148 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "31536000", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how long to retain crashes before pruning them", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_recent_interval": { 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_recent_interval", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "1209600", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "time interval in which to warn about recent crashes", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "dashboard", 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ALERTMANAGER_API_HOST": { 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ALERTMANAGER_API_HOST", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.151 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "AUDIT_API_ENABLED": { 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "AUDIT_API_ENABLED", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "AUDIT_API_LOG_PAYLOAD": { 2024-02-10T09:17:19.153 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "AUDIT_API_LOG_PAYLOAD", 2024-02-10T09:17:19.153 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.153 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.153 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.153 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ENABLE_BROWSABLE_API": { 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ENABLE_BROWSABLE_API", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_cephfs": { 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_cephfs", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.157 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_iscsi": { 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_iscsi", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.158 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_mirroring": { 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_mirroring", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_nfs": { 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_nfs", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_rbd": { 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_rbd", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "FEATURE_TOGGLE_rgw": { 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "FEATURE_TOGGLE_rgw", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.163 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_PASSWORD": { 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_PASSWORD", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "admin", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.164 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_SSL_VERIFY": { 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_SSL_VERIFY", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.165 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_URL": { 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_URL", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_API_USERNAME": { 2024-02-10T09:17:19.166 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_API_USERNAME", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "admin", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "GRAFANA_UPDATE_DASHBOARDS": { 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.167 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ISCSI_API_SSL_VERIFICATION": { 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ISCSI_API_SSL_VERIFICATION", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.168 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PROMETHEUS_API_HOST": { 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PROMETHEUS_API_HOST", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.169 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.170 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-02-10T09:17:19.171 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.172 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.173 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.174 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_ENABLED": { 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_ENABLED", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_EXCLUSION_LIST": { 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_EXCLUSION_LIST", 2024-02-10T09:17:19.176 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_MIN_COMPLEXITY": { 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:17:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "PWD_POLICY_MIN_LENGTH": { 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "PWD_POLICY_MIN_LENGTH", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "REST_REQUESTS_TIMEOUT": { 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "REST_REQUESTS_TIMEOUT", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "45", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_ACCESS_KEY": { 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_ACCESS_KEY", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_ADMIN_RESOURCE": { 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_ADMIN_RESOURCE", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "admin", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_HOST": { 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_HOST", 2024-02-10T09:17:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_PORT": { 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_PORT", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "80", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_SCHEME": { 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_SCHEME", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "http", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_SECRET_KEY": { 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_SECRET_KEY", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.184 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_SSL_VERIFY": { 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_SSL_VERIFY", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.185 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "RGW_API_USER_ID": { 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "RGW_API_USER_ID", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "USER_PWD_EXPIRATION_SPAN": { 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "USER_PWD_EXPIRATION_SPAN", 2024-02-10T09:17:19.186 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "0", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "USER_PWD_EXPIRATION_WARNING_1": { 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.187 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "USER_PWD_EXPIRATION_WARNING_2": { 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:17:19.188 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crt_file": { 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "crt_file", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug": { 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "debug", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Enable/disable debug options", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "jwt_token_ttl": { 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "jwt_token_ttl", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "28800", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "key_file": { 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "key_file", 2024-02-10T09:17:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "password": { 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "password", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_addr": { 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_addr", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "::", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_port": { 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_port", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8080", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssl": { 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssl", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssl_server_port": { 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssl_server_port", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8443", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "standby_behaviour": { 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "standby_behaviour", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "redirect", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "redirect" 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "standby_error_status_code": { 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "standby_error_status_code", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "500", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "400", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "599", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "url_prefix": { 2024-02-10T09:17:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "url_prefix", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "username": { 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "username", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.203 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "devicehealth", 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.204 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enable_monitoring": { 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "enable_monitoring", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "monitor device health metrics", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.205 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.206 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mark_out_threshold": { 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mark_out_threshold", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "2419200", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "automatically mark OSD if it may fail before this long", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pool_name": { 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pool_name", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "device_health_metrics", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "name of pool in which to store device health metrics", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "retention_period": { 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "retention_period", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "15552000", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how long to retain device health metrics", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "scrape_frequency": { 2024-02-10T09:17:19.212 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "scrape_frequency", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "86400", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to scrape device health metrics", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "self_heal": { 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "self_heal", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.213 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "preemptively heal cluster around devices that may fail", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.214 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to wake up and check device health", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warn_threshold": { 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "warn_threshold", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "7257600", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "raise health warning if OSD may fail before this long", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.215 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "diskprediction_local", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.216 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.217 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.218 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.219 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "predict_interval": { 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "predict_interval", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "86400", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.220 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "predictor_model": { 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "predictor_model", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "prophetstor", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.221 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "600", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.222 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "influx", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": false, 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "influxdb python module not found", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "batch_size": { 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "batch_size", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5000", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.223 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "database": { 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "database", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "ceph", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.224 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "hostname": { 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "hostname", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:17:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "30", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.226 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.227 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.229 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "password": { 2024-02-10T09:17:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "password", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "port": { 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "port", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.231 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "8086", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ssl": { 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ssl", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.232 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "false", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "threads": { 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "threads", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.233 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "username": { 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "username", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.234 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "verify_ssl": { 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "verify_ssl", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "true", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.235 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "insights", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.236 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.237 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.238 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.239 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "iostat", 2024-02-10T09:17:19.240 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.241 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.242 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.243 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.244 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "k8sevents", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "ceph_event_retention_days": { 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "ceph_event_retention_days", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.245 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "7", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Days to hold ceph event information within local cache", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "config_check_secs": { 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "config_check_secs", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "10", 2024-02-10T09:17:19.246 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "interval (secs) to check for cluster configuration changes", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.247 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.248 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.250 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "localpool", 2024-02-10T09:17:19.251 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "failure_domain": { 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "failure_domain", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "host", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "failure domain for any created local pool", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.252 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.253 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.254 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.255 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min_size": { 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "min_size", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "default min_size for any created local pool", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "num_rep": { 2024-02-10T09:17:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "num_rep", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "3", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "default replica count for any created local pool", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pg_num": { 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pg_num", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "128", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "default pg_num for any created local pool", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "prefix": { 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "prefix", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "name prefix for any created local pool", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "subtree": { 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "subtree", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "rack", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "CRUSH level for which to create a local pool", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "orchestrator", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.261 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.262 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.264 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "orchestrator": { 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "orchestrator", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.265 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "cephadm", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rook", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "test_orchestrator" 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Orchestrator backend", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.266 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "osd_support", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.267 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.268 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.269 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.270 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "pg_autoscaler", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.271 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.272 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "sleep_interval": { 2024-02-10T09:17:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "sleep_interval", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "progress", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.279 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.280 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max_completed_events": { 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "max_completed_events", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "50", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.281 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "number of past completed events to remember", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "persist_interval": { 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "persist_interval", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "5", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "how frequently to persist completed events", 2024-02-10T09:17:19.282 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "prometheus", 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.283 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.284 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.285 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.287 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_stats_pools": { 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rbd_stats_pools", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.288 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_stats_pools_refresh_interval": { 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rbd_stats_pools_refresh_interval", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.289 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "scrape_interval": { 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "scrape_interval", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_addr": { 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_addr", 2024-02-10T09:17:19.290 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_port": { 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_port", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.291 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rbd_support", 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.292 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.293 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.294 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.295 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.296 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mirror_snapshot_schedule": { 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "mirror_snapshot_schedule", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.297 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "trash_purge_schedule": { 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "trash_purge_schedule", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.298 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "restful", 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "key_file": { 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "key_file", 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.299 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.300 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.301 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.302 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.303 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_addr": { 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_addr", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.305 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "server_port": { 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "server_port", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.306 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rook", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.308 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.309 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.310 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "selftest", 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.313 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.314 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.315 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.316 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "roption1": { 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "roption1", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.317 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "roption2": { 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "roption2", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "xyz", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.318 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption1": { 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption1", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption2": { 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption2", 2024-02-10T09:17:19.319 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption3": { 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption3", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "float", 2024-02-10T09:17:19.320 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption4": { 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption4", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.321 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption5": { 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption5", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rwoption6": { 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "rwoption6", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "testkey": { 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "testkey", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "testlkey": { 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "testlkey", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.325 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "testnewline": { 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "testnewline", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "status", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.331 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "telegraf", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "address": { 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "address", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "unixgram:///tmp/telegraf.sock", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.332 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "15", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.333 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.334 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.335 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "telemetry", 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_basic": { 2024-02-10T09:17:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_basic", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "Share basic cluster information (size, version)", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_crash": { 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_crash", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_device": { 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_device", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "True", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "channel_ident": { 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "channel_ident", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "contact": { 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "contact", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "description": { 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "description", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "device_url": { 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "device_url", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "https://telemetry.ceph.com/device", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enabled": { 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "enabled", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "24", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "8", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "last_opt_revision": { 2024-02-10T09:17:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "last_opt_revision", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "1", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "leaderboard": { 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "leaderboard", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "organization": { 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "organization", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "proxy": { 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "proxy", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "url": { 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "url", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "https://telemetry.ceph.com/report", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "test_orchestrator", 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "volumes", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "can_run": true, 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error_string": "", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "module_options": { 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "discovery_interval": { 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "discovery_interval", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "100", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "identifier": { 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "identifier", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "interval": { 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "interval", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "secs", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "60", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_level": { 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_level", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.372 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster": { 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_cluster_level": { 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_cluster_level", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "info", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [ 2024-02-10T09:17:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "critical", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "debug", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "error", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "info", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "warning" 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "log_to_file": { 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "log_to_file", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "bool", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.375 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 1, 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "False", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "zabbix_host": { 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix_host", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.376 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "None", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "zabbix_port": { 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix_port", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "int", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "10051", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.377 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "zabbix_sender": { 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "name": "zabbix_sender", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "str", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "level": "advanced", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "flags": 0, 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "default_value": "/usr/bin/zabbix_sender", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "min": "", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "max": "", 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "enum_allowed": [], 2024-02-10T09:17:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "desc": "", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "long_desc": "", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "tags": [], 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "see_also": [] 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "services": {}, 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "always_on_modules": { 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nautilus": [ 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "balancer", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crash", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "devicehealth", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "orchestrator_cli", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress", 2024-02-10T09:17:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_support", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "volumes" 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "octopus": [ 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "balancer", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "crash", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "devicehealth", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "orchestrator", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "osd_support", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "pg_autoscaler", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "progress", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "rbd_support", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "status", 2024-02-10T09:17:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "telemetry", 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "volumes" 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ], 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "last_failure_osd_epoch": 4, 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "active_clients": [ 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:0", 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 1486748846 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:0", 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 94314011 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout }, 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addrvec": [ 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "type": "v2", 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "addr": "172.21.15.153:0", 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "nonce": 2708898466 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:17:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout ] 2024-02-10T09:17:19.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:19.383 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:18 smithi153 bash[35780]: audit 2024-02-10T09:17:18.736022+0000 mon.a (mon.0) 70 : audit [DBG] from='client.? 172.21.15.153:0/1359456716' entity='client.admin' cmd=[{"prefix": "mgr dump"}]: dispatch 2024-02-10T09:17:19.734 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for the mgr to restart... 2024-02-10T09:17:19.734 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Waiting for Mgr epoch 9... 2024-02-10T09:17:19.734 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'tell', 'mgr', 'mgr_status'] 2024-02-10T09:17:19.734 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 tell mgr mgr_status 2024-02-10T09:17:20.938 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:20 smithi153 bash[36014]: Warning: Permanently added 'smithi153,172.21.15.153' (ECDSA) to the list of known hosts. 2024-02-10T09:17:21.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout { 2024-02-10T09:17:21.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "mgrmap_epoch": 11, 2024-02-10T09:17:21.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout "initialized": true 2024-02-10T09:17:21.593 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout } 2024-02-10T09:17:21.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: cluster 2024-02-10T09:17:20.528643+0000 mon.a (mon.0) 71 : cluster [INF] Active manager daemon y restarted 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: cluster 2024-02-10T09:17:20.529083+0000 mon.a (mon.0) 72 : cluster [INF] Activating manager daemon y 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: cluster 2024-02-10T09:17:20.531244+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: cluster 2024-02-10T09:17:20.582344+0000 mon.a (mon.0) 74 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0533536s) 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.583669+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.584434+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.584561+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.584672+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.584952+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.585423+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.585601+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:17:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.585769+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.585941+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.586107+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.586269+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: cluster 2024-02-10T09:17:20.599006+0000 mon.a (mon.0) 86 : cluster [INF] Manager daemon y is now available 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:20.621831+0000 mon.a (mon.0) 87 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:21.106293+0000 mon.a (mon.0) 88 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T09:17:21.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:21 smithi153 bash[35780]: audit 2024-02-10T09:17:21.106658+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:17:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Generating a dashboard self-signed certificate... 2024-02-10T09:17:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'dashboard', 'create-self-signed-cert'] 2024-02-10T09:17:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 dashboard create-self-signed-cert 2024-02-10T09:17:22.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:22 smithi153 bash[35780]: cluster 2024-02-10T09:17:21.584802+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e11: y(active, since 1.05581s) 2024-02-10T09:17:23.383 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout Self-signed certificate created 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: cluster 2024-02-10T09:17:22.586402+0000 mon.a (mon.0) 91 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:22.906023+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905437\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905534\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}}, \"devices\": [], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:22.905610\", \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:22.907888+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905437\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905534\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}}, \"devices\": [], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:22.905610\", \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:22.908304+0000 mon.a (mon.0) 94 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:23.379126+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/dashboard/crt","val":"-----BEGIN CERTIFICATE-----\nMIICzzCCAbcCEGvrCzv3h0Z3oRQVgMFNUkcwDQYJKoZIhvcNAQENBQAwJjELMAkG\nA1UECgwCSVQxFzAVBgNVBAMMDmNlcGgtZGFzaGJvYXJkMB4XDTI0MDIxMDA5MTcy\nM1oXDTM0MDIwNzA5MTcyM1owJjELMAkGA1UECgwCSVQxFzAVBgNVBAMMDmNlcGgt\nZGFzaGJvYXJkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqCxYqO55\n8ElJ/UU0Aaxi+46l/gy9b9ZOQ+wBnKBiSlAkPngiF3hy4QalblAK0hJO/FKoqGbv\nYKWez7ISGzp2GtqWyyp055+GuhIej/eM1jMnY5ItHDUH0lnKxaRPbRkHiIWa8GfD\n+XuiPxtDPHb/Sx4J0gKIyPbuJ/1hC34tZQQGKzXZq8l+s9nLwlgp+80mL5d3BAIv\npZf+Lg7U+024s9u4R0FW4/+gfswkvaQsioU7WOSznUMQ095xQhKZ7anyFMkLJxib\nzHCQfJSoNtaepmOOcMII51LWTQ5xJGRzz3nwzacden64xJ59DVlaroa3kpgNFsA7\nf1ALvzsyvjbAYwIDAQABMA0GCSqGSIb3DQEBDQUAA4IBAQADPuLIzqOeHvdCxajw\nIUlmB5YNooLWT+H0ghTqQT0nvuTwTQ7QdXeH0v2jXuYQgm2jTYkskplCt1+qgR7S\nqFE8WtVlITNPjk98pIvI65OTfucd0LfDVh2cW2o1c12fj3ZdrEYZxHJwIJ5zTIMF\nVxh46PLm0ac6bLNlGVBN8Z08btZ1RGv9iqQ81L8PpS8kguTzDn8uj+CinwBM6cXC\nakH0xOOg4ZYeyne7ZDhRQ5XInq4s7pneO6/n0VMOA0cEMvGjikL3UJ1WXecValiX\nkM4Z0JpBLTSQ7LQkfP0Ev0K638/TwlCY2qajf9ouUwT/z3BGUZDql1kEQbp4cCin\nPDr5\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:23.380860+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/dashboard/crt","val":"-----BEGIN CERTIFICATE-----\nMIICzzCCAbcCEGvrCzv3h0Z3oRQVgMFNUkcwDQYJKoZIhvcNAQENBQAwJjELMAkG\nA1UECgwCSVQxFzAVBgNVBAMMDmNlcGgtZGFzaGJvYXJkMB4XDTI0MDIxMDA5MTcy\nM1oXDTM0MDIwNzA5MTcyM1owJjELMAkGA1UECgwCSVQxFzAVBgNVBAMMDmNlcGgt\nZGFzaGJvYXJkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqCxYqO55\n8ElJ/UU0Aaxi+46l/gy9b9ZOQ+wBnKBiSlAkPngiF3hy4QalblAK0hJO/FKoqGbv\nYKWez7ISGzp2GtqWyyp055+GuhIej/eM1jMnY5ItHDUH0lnKxaRPbRkHiIWa8GfD\n+XuiPxtDPHb/Sx4J0gKIyPbuJ/1hC34tZQQGKzXZq8l+s9nLwlgp+80mL5d3BAIv\npZf+Lg7U+024s9u4R0FW4/+gfswkvaQsioU7WOSznUMQ095xQhKZ7anyFMkLJxib\nzHCQfJSoNtaepmOOcMII51LWTQ5xJGRzz3nwzacden64xJ59DVlaroa3kpgNFsA7\nf1ALvzsyvjbAYwIDAQABMA0GCSqGSIb3DQEBDQUAA4IBAQADPuLIzqOeHvdCxajw\nIUlmB5YNooLWT+H0ghTqQT0nvuTwTQ7QdXeH0v2jXuYQgm2jTYkskplCt1+qgR7S\nqFE8WtVlITNPjk98pIvI65OTfucd0LfDVh2cW2o1c12fj3ZdrEYZxHJwIJ5zTIMF\nVxh46PLm0ac6bLNlGVBN8Z08btZ1RGv9iqQ81L8PpS8kguTzDn8uj+CinwBM6cXC\nakH0xOOg4ZYeyne7ZDhRQ5XInq4s7pneO6/n0VMOA0cEMvGjikL3UJ1WXecValiX\nkM4Z0JpBLTSQ7LQkfP0Ev0K638/TwlCY2qajf9ouUwT/z3BGUZDql1kEQbp4cCin\nPDr5\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:23.381191+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/dashboard/key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQCoLFio7nnwSUn9\nRTQBrGL7jqX+DL1v1k5D7AGcoGJKUCQ+eCIXeHLhBqVuUArSEk78UqioZu9gpZ7P\nshIbOnYa2pbLKnTnn4a6Eh6P94zWMydjki0cNQfSWcrFpE9tGQeIhZrwZ8P5e6I/\nG0M8dv9LHgnSAojI9u4n/WELfi1lBAYrNdmryX6z2cvCWCn7zSYvl3cEAi+ll/4u\nDtT7Tbiz27hHQVbj/6B+zCS9pCyKhTtY5LOdQxDT3nFCEpntqfIUyQsnGJvMcJB8\nlKg21p6mY45wwgjnUtZNDnEkZHPPefDNpx16frjEnn0NWVquhreSmA0WwDt/UAu/\nOzK+NsBjAgMBAAECggEBAJmuvO7Tz5coQncy8I2+7U2ZVFPPbKZxH2CuoeJyJqBm\nkFiODAmKxuCg5Djk6J6EoK2xE/Og7YxSjciboDRFvnxX2YmcKO0qOUJ9IHTL8mYk\npRA3GwY16AnwiH8I4pR+VQYLokWjbdHli6VD0rPEwxPwRWUnb7zG+2iMAk0wZ2dx\nRFE7StxGf/4433CJ/uVvXvnNWxK8cIVFSKIjjymjKjKfIqih7qIbuq2q9kDFQEmP\nhn7gsSfiE5KL/5ajmoZkMYpXDBqn3Zx2/YARQDYyzY++yGeEgcaTIorawpRw/n5n\nSYWGlS5wIa72wGUGvTj/KKrMC7Y6yLYy6CygD6Hg0+ECgYEA2ZgpXk5BqG3wQXtB\nkmtCCiBtAVGTEETpJ80bKWZbnzF2fli1xQx75paCB945G7likORGiJ2AL36EN3ac\n2+0z2aZEecAaqBpovYrILrRGUN/Zq1spX0PYo46B1eJb4bTq8UbudDyGOf8h/YMo\n6xyDAsG4jbglOPRgEPk4oGd9IJkCgYEAxdsgiZe+WCWCASvWIFwV5qYJjQbf69zS\n7FXE134yBXIcxst3UfsIvKDoJmogwwJLAiQW9OyUn7Vk0UE518buld527i+aZcn/\n2RHUIiQ6x9POIhSpHSx9Be40UsQKQDlT0E5+e2o/uZnb36hAT21fLp9R6B6N4hhU\nwPECsuKVulsCgYAIWM/tcC8QmPcIbAP96+NipA/KZyivDRWsky+hjO2i07Dp89k1\nL99o3wkp6Bpkt93SFQ+nmfMa302PGxu1TAobJUenK/lZzXmLG9HGCOBRpeShsCOM\nJfwfeg38+Gvx4pDIpKHtD41OmDzjjq55JmTYLDo6JYA4CRfOTuEfNhkUsQKBgQCk\nah9zt0FZhnGt/3uEkIEyla9CHgBdw7cS5YMZf6iZleYL3HBfE7iC50ODCJYI9xLV\nyJ+7EA3BWb9ZlFY41PrWV3OFItGu3FlbTNF3PZwLChs1fu25bIrILtYUIQxTqk/X\nviKk8/nYhd6M0Qf/+OcrmwE5XoqQ458qZhzUAi5iGQKBgQC10Wk6WhpnunoSRn04\n9V+5AjucYXYP9HSmjLx1VJU6NaohmXJzYu3JSlsThSqzb+u6Hk9iQNr17vgJRtl3\n8AE2Az+c9VvM/6M4mNckEWk75dDu/PjMj46mKbu21qqC/1h0ryz0gI7tRAkJ/gvi\nMxK/W5oH3c8o33Mv0T+XNTea8g==\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:17:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:23 smithi153 bash[35780]: audit 2024-02-10T09:17:23.382411+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/dashboard/key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQCoLFio7nnwSUn9\nRTQBrGL7jqX+DL1v1k5D7AGcoGJKUCQ+eCIXeHLhBqVuUArSEk78UqioZu9gpZ7P\nshIbOnYa2pbLKnTnn4a6Eh6P94zWMydjki0cNQfSWcrFpE9tGQeIhZrwZ8P5e6I/\nG0M8dv9LHgnSAojI9u4n/WELfi1lBAYrNdmryX6z2cvCWCn7zSYvl3cEAi+ll/4u\nDtT7Tbiz27hHQVbj/6B+zCS9pCyKhTtY5LOdQxDT3nFCEpntqfIUyQsnGJvMcJB8\nlKg21p6mY45wwgjnUtZNDnEkZHPPefDNpx16frjEnn0NWVquhreSmA0WwDt/UAu/\nOzK+NsBjAgMBAAECggEBAJmuvO7Tz5coQncy8I2+7U2ZVFPPbKZxH2CuoeJyJqBm\nkFiODAmKxuCg5Djk6J6EoK2xE/Og7YxSjciboDRFvnxX2YmcKO0qOUJ9IHTL8mYk\npRA3GwY16AnwiH8I4pR+VQYLokWjbdHli6VD0rPEwxPwRWUnb7zG+2iMAk0wZ2dx\nRFE7StxGf/4433CJ/uVvXvnNWxK8cIVFSKIjjymjKjKfIqih7qIbuq2q9kDFQEmP\nhn7gsSfiE5KL/5ajmoZkMYpXDBqn3Zx2/YARQDYyzY++yGeEgcaTIorawpRw/n5n\nSYWGlS5wIa72wGUGvTj/KKrMC7Y6yLYy6CygD6Hg0+ECgYEA2ZgpXk5BqG3wQXtB\nkmtCCiBtAVGTEETpJ80bKWZbnzF2fli1xQx75paCB945G7likORGiJ2AL36EN3ac\n2+0z2aZEecAaqBpovYrILrRGUN/Zq1spX0PYo46B1eJb4bTq8UbudDyGOf8h/YMo\n6xyDAsG4jbglOPRgEPk4oGd9IJkCgYEAxdsgiZe+WCWCASvWIFwV5qYJjQbf69zS\n7FXE134yBXIcxst3UfsIvKDoJmogwwJLAiQW9OyUn7Vk0UE518buld527i+aZcn/\n2RHUIiQ6x9POIhSpHSx9Be40UsQKQDlT0E5+e2o/uZnb36hAT21fLp9R6B6N4hhU\nwPECsuKVulsCgYAIWM/tcC8QmPcIbAP96+NipA/KZyivDRWsky+hjO2i07Dp89k1\nL99o3wkp6Bpkt93SFQ+nmfMa302PGxu1TAobJUenK/lZzXmLG9HGCOBRpeShsCOM\nJfwfeg38+Gvx4pDIpKHtD41OmDzjjq55JmTYLDo6JYA4CRfOTuEfNhkUsQKBgQCk\nah9zt0FZhnGt/3uEkIEyla9CHgBdw7cS5YMZf6iZleYL3HBfE7iC50ODCJYI9xLV\nyJ+7EA3BWb9ZlFY41PrWV3OFItGu3FlbTNF3PZwLChs1fu25bIrILtYUIQxTqk/X\nviKk8/nYhd6M0Qf/+OcrmwE5XoqQ458qZhzUAi5iGQKBgQC10Wk6WhpnunoSRn04\n9V+5AjucYXYP9HSmjLx1VJU6NaohmXJzYu3JSlsThSqzb+u6Hk9iQNr17vgJRtl3\n8AE2Az+c9VvM/6M4mNckEWk75dDu/PjMj46mKbu21qqC/1h0ryz0gI7tRAkJ/gvi\nMxK/W5oH3c8o33Mv0T+XNTea8g==\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:17:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Creating initial admin user... 2024-02-10T09:17:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'dashboard', 'ac-user-create', 'admin', 'abw6osebun', 'administrator', '--force-password', '--pwd-update-required'] 2024-02-10T09:17:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 dashboard ac-user-create admin abw6osebun administrator --force-password --pwd-update-required 2024-02-10T09:17:24.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:24 smithi153 bash[35780]: audit 2024-02-10T09:17:23.265073+0000 mgr.y (mgr.14140) 1 : audit [DBG] from='client.14153 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout {"username": "admin", "password": "$2b$12$N2bmsIxaytpUNBDJm9J6mua9EQ0LBx8dqPuv0ltuJ3QGHQZt31B.S", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707556645, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T09:17:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:25 smithi153 bash[35780]: audit 2024-02-10T09:17:25.553873+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/dashboard/accessdb_v2","val":"{\"users\": {\"admin\": {\"username\": \"admin\", \"password\": \"$2b$12$N2bmsIxaytpUNBDJm9J6mua9EQ0LBx8dqPuv0ltuJ3QGHQZt31B.S\", \"roles\": [\"administrator\"], \"name\": null, \"email\": null, \"lastUpdate\": 1707556645, \"enabled\": true, \"pwdExpirationDate\": null, \"pwdUpdateRequired\": true}}, \"roles\": {}, \"version\": 2}"}]: dispatch 2024-02-10T09:17:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:25 smithi153 bash[35780]: audit 2024-02-10T09:17:25.555579+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/dashboard/accessdb_v2","val":"{\"users\": {\"admin\": {\"username\": \"admin\", \"password\": \"$2b$12$N2bmsIxaytpUNBDJm9J6mua9EQ0LBx8dqPuv0ltuJ3QGHQZt31B.S\", \"roles\": [\"administrator\"], \"name\": null, \"email\": null, \"lastUpdate\": 1707556645, \"enabled\": true, \"pwdExpirationDate\": null, \"pwdUpdateRequired\": true}}, \"roles\": {}, \"version\": 2}"}]': finished 2024-02-10T09:17:26.197 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Fetching dashboard port number... 2024-02-10T09:17:26.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi153', '-v', '/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z', '-v', '/tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z', '--entrypoint', '/usr/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'get', 'mgr', 'mgr/dashboard/ssl_server_port'] 2024-02-10T09:17:26.197 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Running command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0 -e NODE_NAME=smithi153 -v /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp26w4np5e:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpar9xdpf2:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15.2.0 config get mgr mgr/dashboard/ssl_server_port 2024-02-10T09:17:26.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:26 smithi153 bash[35780]: audit 2024-02-10T09:17:25.331451+0000 mgr.y (mgr.14140) 2 : audit [DBG] from='client.14155 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "password": "abw6osebun", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:/usr/bin/ceph:stdout 8443 2024-02-10T09:17:27.754 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:27 smithi153 bash[35780]: audit 2024-02-10T09:17:27.065531+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:17:27.754 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:27 smithi153 bash[35780]: audit 2024-02-10T09:17:27.247810+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905437\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905534\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:22.905610\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:17:27.754 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:27 smithi153 bash[35780]: audit 2024-02-10T09:17:27.249575+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905437\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:17:22.905534\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:22.905610\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:17:27.754 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:27 smithi153 bash[35780]: audit 2024-02-10T09:17:27.435632+0000 mon.a (mon.0) 104 : audit [DBG] from='client.? 172.21.15.153:0/2590392023' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T09:17:27.754 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:27 smithi153 bash[35780]: cluster 2024-02-10T09:17:27.548627+0000 mon.a (mon.0) 105 : cluster [DBG] mgrmap e13: y(active, since 7s) 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Ceph Dashboard is now available at: 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: URL: https://smithi153.front.sepia.ceph.com:8443/ 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: User: admin 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: Password: abw6osebun 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:You can access the Ceph CLI with: 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.060 INFO:teuthology.orchestra.run.smithi153.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Please consider enabling telemetry to help improve Ceph: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: ceph telemetry on 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr:For more information see: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr:INFO:cephadm:Bootstrap complete. 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Releasing lock 140191482345624 on /run/cephadm/09b43244-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:17:28.061 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG:cephadm:Lock 140191482345624 released on /run/cephadm/09b43244-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:17:28.085 INFO:tasks.cephadm:Fetching config... 2024-02-10T09:17:28.085 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:17:28.085 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T09:17:28.104 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T09:17:28.104 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:17:28.104 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T09:17:28.162 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T09:17:28.162 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:17:28.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T09:17:28.229 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T09:17:28.229 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:17:28.230 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T09:17:28.287 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T09:17:28.287 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T09:17:28.363 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:17:28.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T09:17:28.418 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9O+mZ9XBy2TjO/2KDjnK5x64RlWbXiWcJ8gAwl4VdtC3eFg4EovH+gzqQUhY+kCPMLrZuTIfUqejjS0WFgSS1HqMSy0IJ9UNpJd3q17d7uCh4LZt8gmCvvr8F6XStlavtd72e4+KZbq2kw+Eec3IHeL+DaN02l+Y0XqjHs40sIQgYx9zwwVBiiXAWOmiRKnjC491wNgjotwfdyJ3XmZ0VzdVTtJ1Ka+ziYhv+CewAd0G8KpiC4TZyuJ2lw2YjNPh8KGYSirgrwxo51m0tBLC50kIW/lBvjRrcmlxIJmqSBDoJEmw1wgryjgmfIrPM9tzToQz0Xg4alEsDldxPhSFPCS/UF34UrC+tPXfdHnrQjlLTQDt61XVmwM9bMtQo8PApoTDGSOcRXGmD6pUZA7WDh+HbfuWrg3lnWnxACkO+MtB1dvGMbSkQr4wVV7JfQBheM8yiIUjjWVaUbWbPS1GoHDx42hbGJxYaVh+QMQyB/XvIrfh5804Epj164K3DI0= ceph-09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:17:28.430 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-02-10T09:17:28.431 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:17:28.431 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-02-10T09:17:28.450 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:17:28.450 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T09:17:28.508 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-02-10T09:17:28.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch host add smithi204 2024-02-10T09:17:41.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:41 smithi153 bash[35780]: cluster 2024-02-10T09:17:40.585878+0000 mgr.y (mgr.14140) 3 : cluster [DBG] pgmap v3: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:43.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:43 smithi153 bash[35780]: cluster 2024-02-10T09:17:42.586158+0000 mgr.y (mgr.14140) 4 : cluster [DBG] pgmap v4: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:45.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:45 smithi153 bash[35780]: cluster 2024-02-10T09:17:44.586503+0000 mgr.y (mgr.14140) 5 : cluster [DBG] pgmap v5: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:47.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:47 smithi153 bash[35780]: cluster 2024-02-10T09:17:46.586771+0000 mgr.y (mgr.14140) 6 : cluster [DBG] pgmap v6: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:49.438 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:17:48 smithi153 bash[36014]: Warning: Permanently added 'smithi204,172.21.15.204' (ECDSA) to the list of known hosts. 2024-02-10T09:17:49.918 INFO:teuthology.orchestra.run.smithi204.stdout:Added host 'smithi204' 2024-02-10T09:17:49.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:49 smithi153 bash[35780]: cluster 2024-02-10T09:17:48.587093+0000 mgr.y (mgr.14140) 7 : cluster [DBG] pgmap v7: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:49 smithi153 bash[35780]: audit 2024-02-10T09:17:49.383252+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"smithi153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:17:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:49 smithi153 bash[35780]: audit 2024-02-10T09:17:49.385467+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"smithi153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:17:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:49 smithi153 bash[35780]: audit 2024-02-10T09:17:49.580946+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {}, \"devices\": [], \"daemon_config_deps\": {}, \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:17:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:49 smithi153 bash[35780]: audit 2024-02-10T09:17:49.585493+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {}, \"devices\": [], \"daemon_config_deps\": {}, \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:17:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:49 smithi153 bash[35780]: audit 2024-02-10T09:17:49.586158+0000 mon.a (mon.0) 110 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:17:50.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T09:17:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:50 smithi153 bash[35780]: audit 2024-02-10T09:17:48.914994+0000 mgr.y (mgr.14140) 8 : audit [DBG] from='client.14159 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:17:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:50 smithi153 bash[35780]: cephadm 2024-02-10T09:17:49.385902+0000 mgr.y (mgr.14140) 9 : cephadm [INF] Added host smithi204 2024-02-10T09:17:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:50 smithi153 bash[35780]: audit 2024-02-10T09:17:49.758266+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {}, \"devices\": [], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:49.757944\", \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:17:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:50 smithi153 bash[35780]: audit 2024-02-10T09:17:49.762694+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {}, \"devices\": [], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:49.757944\", \"networks\": {}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:17:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:50 smithi153 bash[35780]: audit 2024-02-10T09:17:49.763258+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:17:52.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:51 smithi153 bash[35780]: cluster 2024-02-10T09:17:50.587376+0000 mgr.y (mgr.14140) 10 : cluster [DBG] pgmap v8: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:53.516 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:17:53.516 INFO:teuthology.orchestra.run.smithi204.stdout:[{"addr": "smithi153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "smithi204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-02-10T09:17:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:53 smithi153 bash[35780]: cluster 2024-02-10T09:17:52.587738+0000 mgr.y (mgr.14140) 11 : cluster [DBG] pgmap v9: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:54.348 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T09:17:54.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T09:17:54.973 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:54 smithi153 bash[35780]: audit 2024-02-10T09:17:53.514886+0000 mgr.y (mgr.14140) 12 : audit [DBG] from='client.14161 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:17:54.973 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:54 smithi153 bash[35780]: audit 2024-02-10T09:17:54.691064+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:17:55.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:55 smithi153 bash[35780]: cluster 2024-02-10T09:17:54.588070+0000 mgr.y (mgr.14140) 13 : cluster [DBG] pgmap v10: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:55.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:55 smithi153 bash[35780]: audit 2024-02-10T09:17:54.870504+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:49.757944\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:17:55.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:55 smithi153 bash[35780]: audit 2024-02-10T09:17:54.872290+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {}, \"last_daemon_update\": \"2024-02-10T09:17:49.757944\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:17:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-02-10T09:17:57.114 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:56 smithi153 bash[35780]: audit 2024-02-10T09:17:56.230910+0000 mon.a (mon.0) 117 : audit [INF] from='client.? 172.21.15.153:0/1615616980' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T09:17:57.416 INFO:tasks.cephadm:Adding mon.c on smithi153 2024-02-10T09:17:57.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add mon 'smithi153:[v2:172.21.15.153:3301,v1:172.21.15.153:6790]=c' 2024-02-10T09:17:58.114 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:57 smithi153 bash[35780]: cluster 2024-02-10T09:17:56.588424+0000 mgr.y (mgr.14140) 14 : cluster [DBG] pgmap v11: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:58.114 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:57 smithi153 bash[35780]: audit 2024-02-10T09:17:56.776415+0000 mon.a (mon.0) 118 : audit [INF] from='client.? 172.21.15.153:0/1615616980' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T09:17:58.114 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:57 smithi153 bash[35780]: cluster 2024-02-10T09:17:56.776465+0000 mon.a (mon.0) 119 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in 2024-02-10T09:17:59.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:59 smithi153 bash[35780]: cluster 2024-02-10T09:17:58.588695+0000 mgr.y (mgr.14140) 15 : cluster [DBG] pgmap v13: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:17:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:59 smithi153 bash[35780]: audit 2024-02-10T09:17:59.239453+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:17:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:59 smithi153 bash[35780]: audit 2024-02-10T09:17:59.240061+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:17:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:17:59 smithi153 bash[35780]: audit 2024-02-10T09:17:59.240633+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2024-02-10T09:18:00.813 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:00 smithi153 bash[35780]: audit 2024-02-10T09:17:59.237854+0000 mgr.y (mgr.14140) 16 : audit [DBG] from='client.14165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi153:[v2:172.21.15.153:3301,v1:172.21.15.153:6790]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:00.814 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:00 smithi153 bash[35780]: cephadm 2024-02-10T09:17:59.240450+0000 mgr.y (mgr.14140) 17 : cephadm [INF] Deploying daemon mon.c on smithi153 2024-02-10T09:18:01.844 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:01 smithi153 bash[35780]: cluster 2024-02-10T09:18:00.588929+0000 mgr.y (mgr.14140) 18 : cluster [DBG] pgmap v14: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:02.244 INFO:teuthology.orchestra.run.smithi153.stdout:Deployed mon.c on host 'smithi153' 2024-02-10T09:18:02.884 DEBUG:teuthology.orchestra.run.smithi153:mon.c> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T09:18:02.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:18:02.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:18:02.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:18:07.601 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:18:07.601 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"09b43244-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:18:02.260724Z","created":"2024-02-10T09:16:39.264924Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0}],"quorum":[0,1]} 2024-02-10T09:18:07.602 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-02-10T09:18:07.626 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:02.262737+0000 mon.a (mon.0) 126 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:07.626 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:02.589273+0000 mgr.y (mgr.14140) 19 : cluster [DBG] pgmap v15: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:07.626 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:04.261337+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:02.262737+0000 mon.a (mon.0) 126 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:02.589273+0000 mgr.y (mgr.14140) 19 : cluster [DBG] pgmap v15: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:04.261337+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: audit 2024-02-10T09:18:04.303789+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303095\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}}, \"last_daemon_update\": \"2024-02-10T09:18:04.303196\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:04.589530+0000 mgr.y (mgr.14140) 20 : cluster [DBG] pgmap v16: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:06.589775+0000 mgr.y (mgr.14140) 21 : cluster [DBG] pgmap v17: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:07.289593+0000 mon.a (mon.0) 128 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:07.294685+0000 mon.a (mon.0) 129 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:07.294771+0000 mon.a (mon.0) 130 : cluster [DBG] fsmap 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:07.294817+0000 mon.a (mon.0) 131 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in 2024-02-10T09:18:07.627 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:07.295112+0000 mon.a (mon.0) 132 : cluster [DBG] mgrmap e13: y(active, since 46s) 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: audit 2024-02-10T09:18:04.303789+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303095\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}}, \"last_daemon_update\": \"2024-02-10T09:18:04.303196\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:04.589530+0000 mgr.y (mgr.14140) 20 : cluster [DBG] pgmap v16: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:06.589775+0000 mgr.y (mgr.14140) 21 : cluster [DBG] pgmap v17: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:07.289593+0000 mon.a (mon.0) 128 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:07.294685+0000 mon.a (mon.0) 129 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:07.294771+0000 mon.a (mon.0) 130 : cluster [DBG] fsmap 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:07.294817+0000 mon.a (mon.0) 131 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:07.295112+0000 mon.a (mon.0) 132 : cluster [DBG] mgrmap e13: y(active, since 46s) 2024-02-10T09:18:07.628 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: audit 2024-02-10T09:18:07.296624+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303095\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}}, \"last_daemon_update\": \"2024-02-10T09:18:04.303196\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:07.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: cluster 2024-02-10T09:18:07.299091+0000 mon.a (mon.0) 134 : cluster [INF] overall HEALTH_OK 2024-02-10T09:18:07.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[39117]: audit 2024-02-10T09:18:07.301401+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303095\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}}, \"last_daemon_update\": \"2024-02-10T09:18:04.303196\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:07.629 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: audit 2024-02-10T09:18:07.296624+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303095\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}}, \"last_daemon_update\": \"2024-02-10T09:18:04.303196\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:07.629 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: cluster 2024-02-10T09:18:07.299091+0000 mon.a (mon.0) 134 : cluster [INF] overall HEALTH_OK 2024-02-10T09:18:07.629 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:07 smithi153 bash[35780]: audit 2024-02-10T09:18:07.301401+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303095\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}}, \"last_daemon_update\": \"2024-02-10T09:18:04.303196\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:08.234 INFO:tasks.cephadm:Adding mon.b on smithi204 2024-02-10T09:18:08.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add mon smithi204:172.21.15.204=b 2024-02-10T09:18:08.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:08 smithi153 bash[35780]: audit 2024-02-10T09:18:07.600669+0000 mon.a (mon.0) 136 : audit [DBG] from='client.? 172.21.15.153:0/1263580393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:18:08.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:08 smithi153 bash[39117]: audit 2024-02-10T09:18:07.600669+0000 mon.a (mon.0) 136 : audit [DBG] from='client.? 172.21.15.153:0/1263580393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:18:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:09 smithi153 bash[35780]: cluster 2024-02-10T09:18:08.590132+0000 mgr.y (mgr.14140) 22 : cluster [DBG] pgmap v18: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:09 smithi153 bash[39117]: cluster 2024-02-10T09:18:08.590132+0000 mgr.y (mgr.14140) 22 : cluster [DBG] pgmap v18: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:10.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:10 smithi153 bash[35780]: audit 2024-02-10T09:18:09.983994+0000 mon.a (mon.0) 137 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:18:10.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:10 smithi153 bash[35780]: audit 2024-02-10T09:18:09.984750+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:10.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:10 smithi153 bash[35780]: audit 2024-02-10T09:18:09.985656+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2024-02-10T09:18:10.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:10 smithi153 bash[39117]: audit 2024-02-10T09:18:09.983994+0000 mon.a (mon.0) 137 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:18:10.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:10 smithi153 bash[39117]: audit 2024-02-10T09:18:09.984750+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:10.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:10 smithi153 bash[39117]: audit 2024-02-10T09:18:09.985656+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2024-02-10T09:18:11.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:11 smithi153 bash[35780]: audit 2024-02-10T09:18:09.982538+0000 mgr.y (mgr.14140) 23 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi204:172.21.15.204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:11.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:11 smithi153 bash[35780]: cephadm 2024-02-10T09:18:09.985299+0000 mgr.y (mgr.14140) 24 : cephadm [INF] Deploying daemon mon.b on smithi204 2024-02-10T09:18:11.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:11 smithi153 bash[35780]: cluster 2024-02-10T09:18:10.590506+0000 mgr.y (mgr.14140) 25 : cluster [DBG] pgmap v19: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:11.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:11 smithi153 bash[39117]: audit 2024-02-10T09:18:09.982538+0000 mgr.y (mgr.14140) 23 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi204:172.21.15.204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:11.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:11 smithi153 bash[39117]: cephadm 2024-02-10T09:18:09.985299+0000 mgr.y (mgr.14140) 24 : cephadm [INF] Deploying daemon mon.b on smithi204 2024-02-10T09:18:11.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:11 smithi153 bash[39117]: cluster 2024-02-10T09:18:10.590506+0000 mgr.y (mgr.14140) 25 : cluster [DBG] pgmap v19: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:13.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:13 smithi153 bash[35780]: cluster 2024-02-10T09:18:12.590917+0000 mgr.y (mgr.14140) 26 : cluster [DBG] pgmap v20: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:13 smithi153 bash[39117]: cluster 2024-02-10T09:18:12.590917+0000 mgr.y (mgr.14140) 26 : cluster [DBG] pgmap v20: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:15 smithi153 bash[35780]: cluster 2024-02-10T09:18:14.591286+0000 mgr.y (mgr.14140) 27 : cluster [DBG] pgmap v21: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:15.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:15 smithi153 bash[39117]: cluster 2024-02-10T09:18:14.591286+0000 mgr.y (mgr.14140) 27 : cluster [DBG] pgmap v21: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:17.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:17 smithi153 bash[35780]: cluster 2024-02-10T09:18:16.591619+0000 mgr.y (mgr.14140) 28 : cluster [DBG] pgmap v22: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:17.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:17 smithi153 bash[39117]: cluster 2024-02-10T09:18:16.591619+0000 mgr.y (mgr.14140) 28 : cluster [DBG] pgmap v22: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:19.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:19 smithi153 bash[35780]: cluster 2024-02-10T09:18:18.591914+0000 mgr.y (mgr.14140) 29 : cluster [DBG] pgmap v23: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:19.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:19 smithi153 bash[39117]: cluster 2024-02-10T09:18:18.591914+0000 mgr.y (mgr.14140) 29 : cluster [DBG] pgmap v23: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:20 smithi153 bash[35780]: cluster 2024-02-10T09:18:20.645458+0000 mon.a (mon.0) 140 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-10T09:18:21.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:20 smithi153 bash[39117]: cluster 2024-02-10T09:18:20.645458+0000 mon.a (mon.0) 140 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-10T09:18:22.002 INFO:teuthology.orchestra.run.smithi204.stdout:Deployed mon.b on host 'smithi204' 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[35780]: cluster 2024-02-10T09:18:20.592219+0000 mgr.y (mgr.14140) 30 : cluster [DBG] pgmap v24: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[35780]: audit 2024-02-10T09:18:21.578713+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[35780]: audit 2024-02-10T09:18:21.579135+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[35780]: audit 2024-02-10T09:18:21.581219+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[39117]: cluster 2024-02-10T09:18:20.592219+0000 mgr.y (mgr.14140) 30 : cluster [DBG] pgmap v24: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[39117]: audit 2024-02-10T09:18:21.578713+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[39117]: audit 2024-02-10T09:18:21.579135+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:21 smithi153 bash[39117]: audit 2024-02-10T09:18:21.581219+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:23.072 DEBUG:teuthology.orchestra.run.smithi204:mon.b> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T09:18:23.075 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T09:18:23.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:18:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:22.592614+0000 mgr.y (mgr.14140) 31 : cluster [DBG] pgmap v25: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:22.768080+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:22.771055+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: audit 2024-02-10T09:18:24.588588+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:24.592973+0000 mgr.y (mgr.14140) 32 : cluster [DBG] pgmap v26: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:24.764873+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T09:18:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:26.593252+0000 mgr.y (mgr.14140) 33 : cluster [DBG] pgmap v27: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.775381+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:22.592614+0000 mgr.y (mgr.14140) 31 : cluster [DBG] pgmap v25: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:22.768080+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:22.771055+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: audit 2024-02-10T09:18:24.588588+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:24.592973+0000 mgr.y (mgr.14140) 32 : cluster [DBG] pgmap v26: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:24.764873+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:26.593252+0000 mgr.y (mgr.14140) 33 : cluster [DBG] pgmap v27: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.775381+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.796614+0000 mon.a (mon.0) 147 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.802123+0000 mon.a (mon.0) 148 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.802209+0000 mon.a (mon.0) 149 : cluster [DBG] fsmap 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.802246+0000 mon.a (mon.0) 150 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.802465+0000 mon.a (mon.0) 151 : cluster [DBG] mgrmap e13: y(active, since 67s) 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: audit 2024-02-10T09:18:27.803197+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: cluster 2024-02-10T09:18:27.806179+0000 mon.a (mon.0) 153 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg inactive 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[35780]: audit 2024-02-10T09:18:27.810129+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.796614+0000 mon.a (mon.0) 147 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.802123+0000 mon.a (mon.0) 148 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.802209+0000 mon.a (mon.0) 149 : cluster [DBG] fsmap 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.802246+0000 mon.a (mon.0) 150 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in 2024-02-10T09:18:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.802465+0000 mon.a (mon.0) 151 : cluster [DBG] mgrmap e13: y(active, since 67s) 2024-02-10T09:18:28.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: audit 2024-02-10T09:18:27.803197+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:28.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: cluster 2024-02-10T09:18:27.806179+0000 mon.a (mon.0) 153 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg inactive 2024-02-10T09:18:28.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:27 smithi153 bash[39117]: audit 2024-02-10T09:18:27.810129+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:22.592614+0000 mgr.y (mgr.14140) 31 : cluster [DBG] pgmap v25: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:22.768080+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:22.771055+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: audit 2024-02-10T09:18:24.588588+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:24.592973+0000 mgr.y (mgr.14140) 32 : cluster [DBG] pgmap v26: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:24.764873+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:26.593252+0000 mgr.y (mgr.14140) 33 : cluster [DBG] pgmap v27: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.775381+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.796614+0000 mon.a (mon.0) 147 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.802123+0000 mon.a (mon.0) 148 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.802209+0000 mon.a (mon.0) 149 : cluster [DBG] fsmap 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.802246+0000 mon.a (mon.0) 150 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in 2024-02-10T09:18:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.802465+0000 mon.a (mon.0) 151 : cluster [DBG] mgrmap e13: y(active, since 67s) 2024-02-10T09:18:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: audit 2024-02-10T09:18:27.803197+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: cluster 2024-02-10T09:18:27.806179+0000 mon.a (mon.0) 153 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg inactive 2024-02-10T09:18:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:27 smithi204 bash[35809]: audit 2024-02-10T09:18:27.810129+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}}, \"last_daemon_update\": \"2024-02-10T09:18:24.588052\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:29.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:28 smithi153 bash[35780]: cluster 2024-02-10T09:18:28.593778+0000 mgr.y (mgr.14140) 34 : cluster [DBG] pgmap v28: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:28 smithi153 bash[39117]: cluster 2024-02-10T09:18:28.593778+0000 mgr.y (mgr.14140) 34 : cluster [DBG] pgmap v28: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:29.239 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-02-10T09:18:29.239 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"09b43244-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:18:22.763983Z","created":"2024-02-10T09:16:39.264924Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0}],"quorum":[0,1,2]} 2024-02-10T09:18:29.240 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-02-10T09:18:29.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:28 smithi204 bash[35809]: cluster 2024-02-10T09:18:28.593778+0000 mgr.y (mgr.14140) 34 : cluster [DBG] pgmap v28: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:29.794 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T09:18:29.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T09:18:30.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:29 smithi153 bash[35780]: audit 2024-02-10T09:18:29.238766+0000 mon.a (mon.0) 155 : audit [DBG] from='client.? 172.21.15.204:0/659629750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:18:30.126 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:29 smithi153 bash[39117]: audit 2024-02-10T09:18:29.238766+0000 mon.a (mon.0) 155 : audit [DBG] from='client.? 172.21.15.204:0/659629750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:18:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:29 smithi204 bash[35809]: audit 2024-02-10T09:18:29.238766+0000 mon.a (mon.0) 155 : audit [DBG] from='client.? 172.21.15.204:0/659629750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:18:31.043 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:30 smithi153 bash[35780]: cluster 2024-02-10T09:18:30.594251+0000 mgr.y (mgr.14140) 35 : cluster [DBG] pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:31.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:30 smithi153 bash[39117]: cluster 2024-02-10T09:18:30.594251+0000 mgr.y (mgr.14140) 35 : cluster [DBG] pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:30 smithi204 bash[35809]: cluster 2024-02-10T09:18:30.594251+0000 mgr.y (mgr.14140) 35 : cluster [DBG] pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:31.622 INFO:teuthology.orchestra.run.smithi153.stdout:# minimal ceph.conf for 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:18:31.623 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-02-10T09:18:31.623 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:18:31.623 INFO:teuthology.orchestra.run.smithi153.stdout: mon_host = [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-02-10T09:18:32.228 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T09:18:32.228 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:18:32.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T09:18:32.259 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:18:32.260 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T09:18:32.328 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:18:32.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T09:18:32.366 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:18:32.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T09:18:32.443 INFO:tasks.cephadm:Adding mgr.y on smithi153 2024-02-10T09:18:32.443 INFO:tasks.cephadm:Adding mgr.x on smithi204 2024-02-10T09:18:32.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi153=y;smithi204=x' 2024-02-10T09:18:32.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:32 smithi153 bash[35780]: audit 2024-02-10T09:18:31.622393+0000 mon.a (mon.0) 156 : audit [DBG] from='client.? 172.21.15.153:0/3048256714' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:32.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:32 smithi153 bash[39117]: audit 2024-02-10T09:18:31.622393+0000 mon.a (mon.0) 156 : audit [DBG] from='client.? 172.21.15.153:0/3048256714' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:32.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:32 smithi204 bash[35809]: audit 2024-02-10T09:18:31.622393+0000 mon.a (mon.0) 156 : audit [DBG] from='client.? 172.21.15.153:0/3048256714' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:33 smithi204 bash[35809]: cluster 2024-02-10T09:18:32.594673+0000 mgr.y (mgr.14140) 36 : cluster [DBG] pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:34.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:33 smithi153 bash[35780]: cluster 2024-02-10T09:18:32.594673+0000 mgr.y (mgr.14140) 36 : cluster [DBG] pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:33 smithi153 bash[39117]: cluster 2024-02-10T09:18:32.594673+0000 mgr.y (mgr.14140) 36 : cluster [DBG] pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:34.317 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-02-10T09:18:34.968 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:34 smithi204 bash[35809]: audit 2024-02-10T09:18:34.312437+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.mgr","val":"{\"created\": \"2024-02-10T09:18:34.312044\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"y\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"x\", \"network\": \"\"}]}, \"service_type\": \"mgr\"}}"}]: dispatch 2024-02-10T09:18:34.968 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:34 smithi204 bash[35809]: audit 2024-02-10T09:18:34.315005+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.mgr","val":"{\"created\": \"2024-02-10T09:18:34.312044\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"y\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"x\", \"network\": \"\"}]}, \"service_type\": \"mgr\"}}"}]': finished 2024-02-10T09:18:34.968 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:34 smithi204 bash[35809]: audit 2024-02-10T09:18:34.316492+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:18:34.968 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:34 smithi204 bash[35809]: audit 2024-02-10T09:18:34.321602+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T09:18:34.968 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:34 smithi204 bash[35809]: audit 2024-02-10T09:18:34.322451+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:34.968 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:34 smithi204 bash[35809]: audit 2024-02-10T09:18:34.323591+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[35780]: audit 2024-02-10T09:18:34.312437+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.mgr","val":"{\"created\": \"2024-02-10T09:18:34.312044\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"y\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"x\", \"network\": \"\"}]}, \"service_type\": \"mgr\"}}"}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[35780]: audit 2024-02-10T09:18:34.315005+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.mgr","val":"{\"created\": \"2024-02-10T09:18:34.312044\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"y\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"x\", \"network\": \"\"}]}, \"service_type\": \"mgr\"}}"}]': finished 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[35780]: audit 2024-02-10T09:18:34.316492+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[35780]: audit 2024-02-10T09:18:34.321602+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[35780]: audit 2024-02-10T09:18:34.322451+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[35780]: audit 2024-02-10T09:18:34.323591+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[39117]: audit 2024-02-10T09:18:34.312437+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.mgr","val":"{\"created\": \"2024-02-10T09:18:34.312044\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"y\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"x\", \"network\": \"\"}]}, \"service_type\": \"mgr\"}}"}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[39117]: audit 2024-02-10T09:18:34.315005+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.mgr","val":"{\"created\": \"2024-02-10T09:18:34.312044\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"y\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"x\", \"network\": \"\"}]}, \"service_type\": \"mgr\"}}"}]': finished 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[39117]: audit 2024-02-10T09:18:34.316492+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[39117]: audit 2024-02-10T09:18:34.321602+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[39117]: audit 2024-02-10T09:18:34.322451+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:34 smithi153 bash[39117]: audit 2024-02-10T09:18:34.323591+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2024-02-10T09:18:35.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:35 smithi204 bash[35809]: audit 2024-02-10T09:18:34.311399+0000 mgr.y (mgr.14140) 37 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi153=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:35.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:35 smithi204 bash[35809]: cephadm 2024-02-10T09:18:34.312030+0000 mgr.y (mgr.14140) 38 : cephadm [INF] Saving service mgr spec with placement count:2 smithi153=y,smithi204=x 2024-02-10T09:18:35.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:35 smithi204 bash[35809]: cephadm 2024-02-10T09:18:34.323126+0000 mgr.y (mgr.14140) 39 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:18:35.933 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:35 smithi204 bash[35809]: cluster 2024-02-10T09:18:34.595011+0000 mgr.y (mgr.14140) 40 : cluster [DBG] pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:35.945 DEBUG:teuthology.orchestra.run.smithi204:mgr.x> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T09:18:35.947 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T09:18:35.948 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:18:35.948 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T09:18:35.970 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T09:18:35.970 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-02-10T09:18:35.981 INFO:journalctl@ceph.mgr.x.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:18:36.031 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T09:18:36.031 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:36.031 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 166406 Links: 1 2024-02-10T09:18:36.032 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:36.032 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:36.032 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-02-10 09:18:01.770165441 +0000 2024-02-10T09:18:36.032 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-02-10 09:14:24.794521198 +0000 2024-02-10T09:18:36.032 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-02-10 09:14:24.794521198 +0000 2024-02-10T09:18:36.032 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-02-10T09:18:36.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T09:18:36.098 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[39117]: audit 2024-02-10T09:18:34.311399+0000 mgr.y (mgr.14140) 37 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi153=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:36.098 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[39117]: cephadm 2024-02-10T09:18:34.312030+0000 mgr.y (mgr.14140) 38 : cephadm [INF] Saving service mgr spec with placement count:2 smithi153=y,smithi204=x 2024-02-10T09:18:36.098 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[39117]: cephadm 2024-02-10T09:18:34.323126+0000 mgr.y (mgr.14140) 39 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:18:36.098 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[39117]: cluster 2024-02-10T09:18:34.595011+0000 mgr.y (mgr.14140) 40 : cluster [DBG] pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:36.098 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[35780]: audit 2024-02-10T09:18:34.311399+0000 mgr.y (mgr.14140) 37 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi153=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:36.098 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[35780]: cephadm 2024-02-10T09:18:34.312030+0000 mgr.y (mgr.14140) 38 : cephadm [INF] Saving service mgr spec with placement count:2 smithi153=y,smithi204=x 2024-02-10T09:18:36.099 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[35780]: cephadm 2024-02-10T09:18:34.323126+0000 mgr.y (mgr.14140) 39 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:18:36.099 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:35 smithi153 bash[35780]: cluster 2024-02-10T09:18:34.595011+0000 mgr.y (mgr.14140) 40 : cluster [DBG] pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:36.100 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-02-10T09:18:36.100 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-02-10T09:18:36.100 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000246596 s, 2.1 MB/s 2024-02-10T09:18:36.101 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T09:18:36.161 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 166461 Links: 1 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-02-10 09:18:01.770165441 +0000 2024-02-10T09:18:36.220 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-02-10 09:14:25.133515246 +0000 2024-02-10T09:18:36.221 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-02-10 09:14:25.133515246 +0000 2024-02-10T09:18:36.221 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-02-10T09:18:36.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T09:18:36.288 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-02-10T09:18:36.288 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-02-10T09:18:36.288 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000248342 s, 2.1 MB/s 2024-02-10T09:18:36.289 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T09:18:36.348 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 165717 Links: 1 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-02-10 09:18:01.770165441 +0000 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-02-10 09:14:25.475509241 +0000 2024-02-10T09:18:36.408 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-02-10 09:14:25.475509241 +0000 2024-02-10T09:18:36.409 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-02-10T09:18:36.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T09:18:36.474 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-02-10T09:18:36.474 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-02-10T09:18:36.474 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000177342 s, 2.9 MB/s 2024-02-10T09:18:36.475 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T09:18:36.534 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-02-10T09:18:36.592 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T09:18:36.592 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:36.592 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 167545 Links: 1 2024-02-10T09:18:36.592 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:36.593 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:36.593 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-02-10 09:18:01.770165441 +0000 2024-02-10T09:18:36.593 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-02-10 09:14:25.816503253 +0000 2024-02-10T09:18:36.593 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-02-10 09:14:25.816503253 +0000 2024-02-10T09:18:36.593 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-02-10T09:18:36.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T09:18:36.657 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-02-10T09:18:36.657 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-02-10T09:18:36.657 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000209838 s, 2.4 MB/s 2024-02-10T09:18:36.658 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T09:18:36.717 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:18:36.718 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T09:18:36.736 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T09:18:36.736 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-02-10T09:18:36.794 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T09:18:36.794 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:36.794 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 169044 Links: 1 2024-02-10T09:18:36.794 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:36.794 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:36.794 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-02-10 09:18:35.775905316 +0000 2024-02-10T09:18:36.795 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-02-10 09:14:32.576930199 +0000 2024-02-10T09:18:36.795 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-02-10 09:14:32.576930199 +0000 2024-02-10T09:18:36.795 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-02-10T09:18:36.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T09:18:36.864 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-02-10T09:18:36.864 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-02-10T09:18:36.864 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000203766 s, 2.5 MB/s 2024-02-10T09:18:36.865 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T09:18:36.924 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 170238 Links: 1 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-02-10 09:18:35.775905316 +0000 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-02-10 09:14:32.928925387 +0000 2024-02-10T09:18:36.982 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-02-10 09:14:32.928925387 +0000 2024-02-10T09:18:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-02-10T09:18:36.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T09:18:37.047 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:36 smithi204 bash[35809]: audit 2024-02-10T09:18:35.992823+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:37.047 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:36 smithi204 bash[35809]: audit 2024-02-10T09:18:35.997877+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:37.047 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:36 smithi204 bash[35809]: audit 2024-02-10T09:18:35.998502+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:37.049 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-02-10T09:18:37.049 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-02-10T09:18:37.049 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000211906 s, 2.4 MB/s 2024-02-10T09:18:37.050 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T09:18:37.109 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 167120 Links: 1 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-02-10 09:18:35.775905316 +0000 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-02-10 09:14:33.260920849 +0000 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-02-10 09:14:33.260920849 +0000 2024-02-10T09:18:37.168 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-02-10T09:18:37.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T09:18:37.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:36 smithi153 bash[35780]: audit 2024-02-10T09:18:35.992823+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:37.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:36 smithi153 bash[35780]: audit 2024-02-10T09:18:35.997877+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:36 smithi153 bash[35780]: audit 2024-02-10T09:18:35.998502+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:36 smithi153 bash[39117]: audit 2024-02-10T09:18:35.992823+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:36 smithi153 bash[39117]: audit 2024-02-10T09:18:35.997877+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:24.587911\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:36 smithi153 bash[39117]: audit 2024-02-10T09:18:35.998502+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:37.234 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-02-10T09:18:37.234 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-02-10T09:18:37.234 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000189219 s, 2.7 MB/s 2024-02-10T09:18:37.235 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T09:18:37.294 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 164407 Links: 1 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-02-10 09:18:35.775905316 +0000 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-02-10 09:14:33.602916174 +0000 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-02-10 09:14:33.602916174 +0000 2024-02-10T09:18:37.353 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-02-10T09:18:37.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T09:18:37.419 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-02-10T09:18:37.419 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-02-10T09:18:37.419 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000223237 s, 2.3 MB/s 2024-02-10T09:18:37.420 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T09:18:37.489 INFO:tasks.cephadm:Deploying osd.0 on smithi153 with /dev/vg_nvme/lv_4... 2024-02-10T09:18:37.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T09:18:37.840 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[39117]: cluster 2024-02-10T09:18:36.595268+0000 mgr.y (mgr.14140) 41 : cluster [DBG] pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:37.840 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[39117]: audit 2024-02-10T09:18:37.157869+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_daemon_update\": \"2024-02-10T09:18:37.157413\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:37.840 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[35780]: cluster 2024-02-10T09:18:36.595268+0000 mgr.y (mgr.14140) 41 : cluster [DBG] pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:37.840 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[35780]: audit 2024-02-10T09:18:37.157869+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_daemon_update\": \"2024-02-10T09:18:37.157413\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[39117]: audit 2024-02-10T09:18:37.160087+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_daemon_update\": \"2024-02-10T09:18:37.157413\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[39117]: audit 2024-02-10T09:18:37.161286+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[39117]: audit 2024-02-10T09:18:37.161881+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[39117]: audit 2024-02-10T09:18:37.162646+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[35780]: audit 2024-02-10T09:18:37.160087+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_daemon_update\": \"2024-02-10T09:18:37.157413\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[35780]: audit 2024-02-10T09:18:37.161286+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2024-02-10T09:18:37.841 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[35780]: audit 2024-02-10T09:18:37.161881+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:37.842 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:37 smithi153 bash[35780]: audit 2024-02-10T09:18:37.162646+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2024-02-10T09:18:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:37 smithi204 bash[35809]: cluster 2024-02-10T09:18:36.595268+0000 mgr.y (mgr.14140) 41 : cluster [DBG] pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:37 smithi204 bash[35809]: audit 2024-02-10T09:18:37.157869+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_daemon_update\": \"2024-02-10T09:18:37.157413\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:18:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:37 smithi204 bash[35809]: audit 2024-02-10T09:18:37.160087+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}}, \"last_daemon_update\": \"2024-02-10T09:18:37.157413\", \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:18:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:37 smithi204 bash[35809]: audit 2024-02-10T09:18:37.161286+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2024-02-10T09:18:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:37 smithi204 bash[35809]: audit 2024-02-10T09:18:37.161881+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:37 smithi204 bash[35809]: audit 2024-02-10T09:18:37.162646+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2024-02-10T09:18:38.764 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[35780]: cephadm 2024-02-10T09:18:37.161016+0000 mgr.y (mgr.14140) 42 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-10T09:18:39.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:38 smithi204 bash[35809]: cephadm 2024-02-10T09:18:37.161016+0000 mgr.y (mgr.14140) 42 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-10T09:18:39.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:38 smithi204 bash[35809]: cephadm 2024-02-10T09:18:37.162323+0000 mgr.y (mgr.14140) 43 : cephadm [INF] Reconfiguring daemon mgr.y on smithi153 2024-02-10T09:18:39.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:38 smithi204 bash[35809]: audit 2024-02-10T09:18:38.627896+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:39.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:38 smithi204 bash[35809]: audit 2024-02-10T09:18:38.630268+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:39.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:38 smithi204 bash[35809]: audit 2024-02-10T09:18:38.630859+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:39.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[35780]: cephadm 2024-02-10T09:18:37.162323+0000 mgr.y (mgr.14140) 43 : cephadm [INF] Reconfiguring daemon mgr.y on smithi153 2024-02-10T09:18:39.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[35780]: audit 2024-02-10T09:18:38.627896+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:39.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[35780]: audit 2024-02-10T09:18:38.630268+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:39.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[35780]: audit 2024-02-10T09:18:38.630859+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:39.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[39117]: cephadm 2024-02-10T09:18:37.161016+0000 mgr.y (mgr.14140) 42 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-10T09:18:39.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[39117]: cephadm 2024-02-10T09:18:37.162323+0000 mgr.y (mgr.14140) 43 : cephadm [INF] Reconfiguring daemon mgr.y on smithi153 2024-02-10T09:18:39.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[39117]: audit 2024-02-10T09:18:38.627896+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:39.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[39117]: audit 2024-02-10T09:18:38.630268+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303002\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:04.303126\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:39.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:38 smithi153 bash[39117]: audit 2024-02-10T09:18:38.630859+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:39.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:39 smithi153 bash[35780]: cluster 2024-02-10T09:18:38.595682+0000 mgr.y (mgr.14140) 44 : cluster [DBG] pgmap v33: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:39.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:39 smithi153 bash[39117]: cluster 2024-02-10T09:18:38.595682+0000 mgr.y (mgr.14140) 44 : cluster [DBG] pgmap v33: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:39 smithi204 bash[35809]: cluster 2024-02-10T09:18:38.595682+0000 mgr.y (mgr.14140) 44 : cluster [DBG] pgmap v33: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T09:18:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T09:18:40.225 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-02-10T09:18:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-02-10T09:18:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113912 s, 921 MB/s 2024-02-10T09:18:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-02-10T09:18:41.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:40 smithi153 bash[35780]: cluster 2024-02-10T09:18:40.172675+0000 mon.a (mon.0) 174 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:18:41.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:40 smithi153 bash[35780]: audit 2024-02-10T09:18:40.173626+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.204:0/1040329981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:18:41.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:40 smithi153 bash[35780]: audit 2024-02-10T09:18:40.174728+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.204:0/1040329981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:18:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:40 smithi153 bash[39117]: cluster 2024-02-10T09:18:40.172675+0000 mon.a (mon.0) 174 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:18:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:40 smithi153 bash[39117]: audit 2024-02-10T09:18:40.173626+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.204:0/1040329981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:18:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:40 smithi153 bash[39117]: audit 2024-02-10T09:18:40.174728+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.204:0/1040329981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:18:41.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-02-10T09:18:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:40 smithi204 bash[35809]: cluster 2024-02-10T09:18:40.172675+0000 mon.a (mon.0) 174 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:18:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:40 smithi204 bash[35809]: audit 2024-02-10T09:18:40.173626+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.204:0/1040329981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:18:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:40 smithi204 bash[35809]: audit 2024-02-10T09:18:40.174728+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.204:0/1040329981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:18:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:41 smithi204 bash[35809]: cluster 2024-02-10T09:18:40.595968+0000 mgr.y (mgr.14140) 45 : cluster [DBG] pgmap v34: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:41 smithi204 bash[35809]: cluster 2024-02-10T09:18:40.773953+0000 mon.a (mon.0) 175 : cluster [DBG] mgrmap e14: y(active, since 80s), standbys: x 2024-02-10T09:18:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:41 smithi204 bash[35809]: audit 2024-02-10T09:18:40.774059+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:18:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:41 smithi204 bash[35809]: audit 2024-02-10T09:18:41.165430+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_daemon_update\": \"2024-02-10T09:18:41.164850\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:41 smithi204 bash[35809]: audit 2024-02-10T09:18:41.168840+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_daemon_update\": \"2024-02-10T09:18:41.164850\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[35780]: cluster 2024-02-10T09:18:40.595968+0000 mgr.y (mgr.14140) 45 : cluster [DBG] pgmap v34: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[35780]: cluster 2024-02-10T09:18:40.773953+0000 mon.a (mon.0) 175 : cluster [DBG] mgrmap e14: y(active, since 80s), standbys: x 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[35780]: audit 2024-02-10T09:18:40.774059+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[35780]: audit 2024-02-10T09:18:41.165430+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_daemon_update\": \"2024-02-10T09:18:41.164850\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[35780]: audit 2024-02-10T09:18:41.168840+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_daemon_update\": \"2024-02-10T09:18:41.164850\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[39117]: cluster 2024-02-10T09:18:40.595968+0000 mgr.y (mgr.14140) 45 : cluster [DBG] pgmap v34: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[39117]: cluster 2024-02-10T09:18:40.773953+0000 mon.a (mon.0) 175 : cluster [DBG] mgrmap e14: y(active, since 80s), standbys: x 2024-02-10T09:18:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[39117]: audit 2024-02-10T09:18:40.774059+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:18:42.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[39117]: audit 2024-02-10T09:18:41.165430+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_daemon_update\": \"2024-02-10T09:18:41.164850\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:42.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:41 smithi153 bash[39117]: audit 2024-02-10T09:18:41.168840+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}}, \"last_daemon_update\": \"2024-02-10T09:18:41.164850\", \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[35780]: cluster 2024-02-10T09:18:42.596288+0000 mgr.y (mgr.14140) 46 : cluster [DBG] pgmap v35: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[35780]: audit 2024-02-10T09:18:42.863294+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[35780]: audit 2024-02-10T09:18:42.863817+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[35780]: audit 2024-02-10T09:18:42.864410+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[39117]: cluster 2024-02-10T09:18:42.596288+0000 mgr.y (mgr.14140) 46 : cluster [DBG] pgmap v35: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[39117]: audit 2024-02-10T09:18:42.863294+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[39117]: audit 2024-02-10T09:18:42.863817+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:44.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:43 smithi153 bash[39117]: audit 2024-02-10T09:18:42.864410+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:43 smithi204 bash[35809]: cluster 2024-02-10T09:18:42.596288+0000 mgr.y (mgr.14140) 46 : cluster [DBG] pgmap v35: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:43 smithi204 bash[35809]: audit 2024-02-10T09:18:42.863294+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:18:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:43 smithi204 bash[35809]: audit 2024-02-10T09:18:42.863817+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:43 smithi204 bash[35809]: audit 2024-02-10T09:18:42.864410+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:45.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:44 smithi204 bash[35809]: audit 2024-02-10T09:18:42.862308+0000 mgr.y (mgr.14140) 47 : audit [DBG] from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:45.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:44 smithi204 bash[35809]: cephadm 2024-02-10T09:18:42.863103+0000 mgr.y (mgr.14140) 48 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:18:45.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:44 smithi153 bash[35780]: audit 2024-02-10T09:18:42.862308+0000 mgr.y (mgr.14140) 47 : audit [DBG] from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:45.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:44 smithi153 bash[35780]: cephadm 2024-02-10T09:18:42.863103+0000 mgr.y (mgr.14140) 48 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:18:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:44 smithi153 bash[39117]: audit 2024-02-10T09:18:42.862308+0000 mgr.y (mgr.14140) 47 : audit [DBG] from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:44 smithi153 bash[39117]: cephadm 2024-02-10T09:18:42.863103+0000 mgr.y (mgr.14140) 48 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:18:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:45 smithi204 bash[35809]: cluster 2024-02-10T09:18:44.596554+0000 mgr.y (mgr.14140) 49 : cluster [DBG] pgmap v36: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:45 smithi204 bash[35809]: audit 2024-02-10T09:18:45.003740+0000 mon.a (mon.0) 182 : audit [INF] from='client.? 172.21.15.153:0/3580245620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8351789c-a971-44a1-aa2f-b16e87fb72a8"}]: dispatch 2024-02-10T09:18:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:45 smithi204 bash[35809]: audit 2024-02-10T09:18:45.007044+0000 mon.a (mon.0) 183 : audit [INF] from='client.? 172.21.15.153:0/3580245620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8351789c-a971-44a1-aa2f-b16e87fb72a8"}]': finished 2024-02-10T09:18:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:45 smithi204 bash[35809]: cluster 2024-02-10T09:18:45.007080+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e7: 1 total, 0 up, 0 in 2024-02-10T09:18:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:45 smithi204 bash[35809]: audit 2024-02-10T09:18:45.007195+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:18:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:45 smithi204 bash[35809]: audit 2024-02-10T09:18:45.752885+0000 mon.a (mon.0) 186 : audit [DBG] from='client.? 172.21.15.153:0/3677385286' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[35780]: cluster 2024-02-10T09:18:44.596554+0000 mgr.y (mgr.14140) 49 : cluster [DBG] pgmap v36: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[35780]: audit 2024-02-10T09:18:45.003740+0000 mon.a (mon.0) 182 : audit [INF] from='client.? 172.21.15.153:0/3580245620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8351789c-a971-44a1-aa2f-b16e87fb72a8"}]: dispatch 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[35780]: audit 2024-02-10T09:18:45.007044+0000 mon.a (mon.0) 183 : audit [INF] from='client.? 172.21.15.153:0/3580245620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8351789c-a971-44a1-aa2f-b16e87fb72a8"}]': finished 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[35780]: cluster 2024-02-10T09:18:45.007080+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e7: 1 total, 0 up, 0 in 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[35780]: audit 2024-02-10T09:18:45.007195+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[35780]: audit 2024-02-10T09:18:45.752885+0000 mon.a (mon.0) 186 : audit [DBG] from='client.? 172.21.15.153:0/3677385286' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[39117]: cluster 2024-02-10T09:18:44.596554+0000 mgr.y (mgr.14140) 49 : cluster [DBG] pgmap v36: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[39117]: audit 2024-02-10T09:18:45.003740+0000 mon.a (mon.0) 182 : audit [INF] from='client.? 172.21.15.153:0/3580245620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8351789c-a971-44a1-aa2f-b16e87fb72a8"}]: dispatch 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[39117]: audit 2024-02-10T09:18:45.007044+0000 mon.a (mon.0) 183 : audit [INF] from='client.? 172.21.15.153:0/3580245620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8351789c-a971-44a1-aa2f-b16e87fb72a8"}]': finished 2024-02-10T09:18:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[39117]: cluster 2024-02-10T09:18:45.007080+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e7: 1 total, 0 up, 0 in 2024-02-10T09:18:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[39117]: audit 2024-02-10T09:18:45.007195+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:18:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:45 smithi153 bash[39117]: audit 2024-02-10T09:18:45.752885+0000 mon.a (mon.0) 186 : audit [DBG] from='client.? 172.21.15.153:0/3677385286' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:18:47.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:47 smithi204 bash[35809]: cluster 2024-02-10T09:18:46.596962+0000 mgr.y (mgr.14140) 50 : cluster [DBG] pgmap v38: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:47.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:47 smithi153 bash[35780]: cluster 2024-02-10T09:18:46.596962+0000 mgr.y (mgr.14140) 50 : cluster [DBG] pgmap v38: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:47.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:47 smithi153 bash[39117]: cluster 2024-02-10T09:18:46.596962+0000 mgr.y (mgr.14140) 50 : cluster [DBG] pgmap v38: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:48 smithi204 bash[35809]: audit 2024-02-10T09:18:48.257979+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:48.836 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:48 smithi153 bash[35780]: audit 2024-02-10T09:18:48.257979+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:48.836 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:48 smithi153 bash[39117]: audit 2024-02-10T09:18:48.257979+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:49.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:49 smithi153 bash[35780]: cluster 2024-02-10T09:18:48.597349+0000 mgr.y (mgr.14140) 51 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:49.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:49 smithi153 bash[39117]: cluster 2024-02-10T09:18:48.597349+0000 mgr.y (mgr.14140) 51 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:49.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:49 smithi204 bash[35809]: cluster 2024-02-10T09:18:48.597349+0000 mgr.y (mgr.14140) 51 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:50.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:50 smithi153 bash[35780]: audit 2024-02-10T09:18:50.609702+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:18:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:50 smithi153 bash[35780]: audit 2024-02-10T09:18:50.610274+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:50 smithi153 bash[35780]: audit 2024-02-10T09:18:50.610836+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2024-02-10T09:18:50.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:50 smithi153 bash[39117]: audit 2024-02-10T09:18:50.609702+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:18:50.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:50 smithi153 bash[39117]: audit 2024-02-10T09:18:50.610274+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:50.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:50 smithi153 bash[39117]: audit 2024-02-10T09:18:50.610836+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2024-02-10T09:18:51.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:50 smithi204 bash[35809]: audit 2024-02-10T09:18:50.609702+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:18:51.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:50 smithi204 bash[35809]: audit 2024-02-10T09:18:50.610274+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:51.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:50 smithi204 bash[35809]: audit 2024-02-10T09:18:50.610836+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2024-02-10T09:18:51.872 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:51 smithi153 bash[35780]: cluster 2024-02-10T09:18:50.597655+0000 mgr.y (mgr.14140) 52 : cluster [DBG] pgmap v40: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:51.872 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:51 smithi153 bash[35780]: cephadm 2024-02-10T09:18:50.610650+0000 mgr.y (mgr.14140) 53 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:18:51.872 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:51 smithi153 bash[39117]: cluster 2024-02-10T09:18:50.597655+0000 mgr.y (mgr.14140) 52 : cluster [DBG] pgmap v40: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:51.872 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:51 smithi153 bash[39117]: cephadm 2024-02-10T09:18:50.610650+0000 mgr.y (mgr.14140) 53 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:18:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:51 smithi204 bash[35809]: cluster 2024-02-10T09:18:50.597655+0000 mgr.y (mgr.14140) 52 : cluster [DBG] pgmap v40: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:51 smithi204 bash[35809]: cephadm 2024-02-10T09:18:50.610650+0000 mgr.y (mgr.14140) 53 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:18:52.180 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 0 on host 'smithi153' 2024-02-10T09:18:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:52 smithi153 bash[35780]: audit 2024-02-10T09:18:52.174421+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:52 smithi153 bash[39117]: audit 2024-02-10T09:18:52.174421+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:52 smithi153 bash[35780]: audit 2024-02-10T09:18:52.174965+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:52 smithi153 bash[35780]: audit 2024-02-10T09:18:52.177221+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:52.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:52 smithi153 bash[39117]: audit 2024-02-10T09:18:52.174965+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:52.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:52 smithi153 bash[39117]: audit 2024-02-10T09:18:52.177221+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:52.972 DEBUG:teuthology.orchestra.run.smithi153:osd.0> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T09:18:52.974 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_3... 2024-02-10T09:18:52.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T09:18:53.003 INFO:journalctl@ceph.osd.0.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:18:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:52 smithi204 bash[35809]: audit 2024-02-10T09:18:52.174421+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:52 smithi204 bash[35809]: audit 2024-02-10T09:18:52.174965+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:18:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:52 smithi204 bash[35809]: audit 2024-02-10T09:18:52.177221+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164645\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164719\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:41.164778\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_device_update\": \"2024-02-10T09:17:27.247281\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:53.669 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T09:18:53.669 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-02-10T09:18:53.669 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T09:18:53.669 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-02-10T09:18:53.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T09:18:53.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T09:18:53.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[41172]: --> ceph-volume lvm activate successful for osd ID: 0 2024-02-10T09:18:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[35780]: cluster 2024-02-10T09:18:52.598101+0000 mgr.y (mgr.14140) 54 : cluster [DBG] pgmap v41: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:53.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:53 smithi153 bash[39117]: cluster 2024-02-10T09:18:52.598101+0000 mgr.y (mgr.14140) 54 : cluster [DBG] pgmap v41: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:53 smithi204 bash[35809]: cluster 2024-02-10T09:18:52.598101+0000 mgr.y (mgr.14140) 54 : cluster [DBG] pgmap v41: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T09:18:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T09:18:55.337 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-02-10T09:18:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-02-10T09:18:55.337 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113435 s, 924 MB/s 2024-02-10T09:18:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-02-10T09:18:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:55 smithi153 bash[35780]: cluster 2024-02-10T09:18:54.598471+0000 mgr.y (mgr.14140) 55 : cluster [DBG] pgmap v42: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:55.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:55 smithi153 bash[39117]: cluster 2024-02-10T09:18:54.598471+0000 mgr.y (mgr.14140) 55 : cluster [DBG] pgmap v42: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:56.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:55 smithi204 bash[35809]: cluster 2024-02-10T09:18:54.598471+0000 mgr.y (mgr.14140) 55 : cluster [DBG] pgmap v42: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:56.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-02-10T09:18:57.368 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[41523]: debug 2024-02-10T09:18:57.355+0000 7f5f6702fec0 -1 Falling back to public interface 2024-02-10T09:18:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[35780]: cluster 2024-02-10T09:18:56.598823+0000 mgr.y (mgr.14140) 56 : cluster [DBG] pgmap v43: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[39117]: cluster 2024-02-10T09:18:56.598823+0000 mgr.y (mgr.14140) 56 : cluster [DBG] pgmap v43: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[35780]: audit 2024-02-10T09:18:56.958848+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[35780]: audit 2024-02-10T09:18:56.963801+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[35780]: audit 2024-02-10T09:18:56.964334+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:57.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[39117]: audit 2024-02-10T09:18:56.958848+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:57.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[39117]: audit 2024-02-10T09:18:56.963801+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:57.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[39117]: audit 2024-02-10T09:18:56.964334+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:57.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:57 smithi204 bash[35809]: cluster 2024-02-10T09:18:56.598823+0000 mgr.y (mgr.14140) 56 : cluster [DBG] pgmap v43: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:57.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:57 smithi204 bash[35809]: audit 2024-02-10T09:18:56.958848+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:18:57.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:57 smithi204 bash[35809]: audit 2024-02-10T09:18:56.963801+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:18:57.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:57 smithi204 bash[35809]: audit 2024-02-10T09:18:56.964334+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:58.143 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:57 smithi153 bash[41523]: debug 2024-02-10T09:18:57.905+0000 7f5f6702fec0 -1 osd.0 0 log_to_monitors {default=true} 2024-02-10T09:18:58.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:58 smithi204 bash[35809]: audit 2024-02-10T09:18:57.908425+0000 mon.a (mon.0) 197 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:18:58.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:58 smithi153 bash[35780]: audit 2024-02-10T09:18:57.908425+0000 mon.a (mon.0) 197 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:18:58.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:58 smithi153 bash[39117]: audit 2024-02-10T09:18:57.908425+0000 mon.a (mon.0) 197 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:18:59.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[41523]: debug 2024-02-10T09:18:59.575+0000 7f5f4bddc700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T09:18:59.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[41523]: debug 2024-02-10T09:18:59.580+0000 7f5f56ff8700 -1 osd.0 9 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.572777+0000 mon.a (mon.0) 198 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: cluster 2024-02-10T09:18:58.572825+0000 mon.a (mon.0) 199 : cluster [DBG] osdmap e8: 1 total, 0 up, 0 in 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.572933+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.573284+0000 mon.a (mon.0) 201 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: cluster 2024-02-10T09:18:58.599034+0000 mgr.y (mgr.14140) 57 : cluster [DBG] pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.729312+0000 mgr.y (mgr.14140) 58 : audit [DBG] from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: cephadm 2024-02-10T09:18:58.729957+0000 mgr.y (mgr.14140) 59 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.730215+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.730717+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: audit 2024-02-10T09:18:58.731339+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: cluster 2024-02-10T09:18:58.903431+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[35780]: cluster 2024-02-10T09:18:58.903495+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.572777+0000 mon.a (mon.0) 198 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: cluster 2024-02-10T09:18:58.572825+0000 mon.a (mon.0) 199 : cluster [DBG] osdmap e8: 1 total, 0 up, 0 in 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.572933+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.573284+0000 mon.a (mon.0) 201 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: cluster 2024-02-10T09:18:58.599034+0000 mgr.y (mgr.14140) 57 : cluster [DBG] pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.729312+0000 mgr.y (mgr.14140) 58 : audit [DBG] from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:59.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: cephadm 2024-02-10T09:18:58.729957+0000 mgr.y (mgr.14140) 59 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:18:59.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.730215+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:18:59.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.730717+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:59.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: audit 2024-02-10T09:18:58.731339+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:59.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: cluster 2024-02-10T09:18:58.903431+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:18:59.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:18:59 smithi153 bash[39117]: cluster 2024-02-10T09:18:58.903495+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.572777+0000 mon.a (mon.0) 198 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: cluster 2024-02-10T09:18:58.572825+0000 mon.a (mon.0) 199 : cluster [DBG] osdmap e8: 1 total, 0 up, 0 in 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.572933+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.573284+0000 mon.a (mon.0) 201 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: cluster 2024-02-10T09:18:58.599034+0000 mgr.y (mgr.14140) 57 : cluster [DBG] pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.729312+0000 mgr.y (mgr.14140) 58 : audit [DBG] from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: cephadm 2024-02-10T09:18:58.729957+0000 mgr.y (mgr.14140) 59 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.730215+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.730717+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:18:59.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: audit 2024-02-10T09:18:58.731339+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:18:59.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: cluster 2024-02-10T09:18:58.903431+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:18:59.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:18:59 smithi204 bash[35809]: cluster 2024-02-10T09:18:58.903495+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:00 smithi153 bash[35780]: audit 2024-02-10T09:18:59.573552+0000 mon.a (mon.0) 205 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:00 smithi153 bash[35780]: cluster 2024-02-10T09:18:59.573586+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e9: 1 total, 0 up, 0 in 2024-02-10T09:19:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:00 smithi153 bash[35780]: audit 2024-02-10T09:18:59.574057+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:00.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:00 smithi153 bash[39117]: audit 2024-02-10T09:18:59.573552+0000 mon.a (mon.0) 205 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:00.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:00 smithi153 bash[39117]: cluster 2024-02-10T09:18:59.573586+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e9: 1 total, 0 up, 0 in 2024-02-10T09:19:00.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:00 smithi153 bash[39117]: audit 2024-02-10T09:18:59.574057+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:01.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:00 smithi204 bash[35809]: audit 2024-02-10T09:18:59.573552+0000 mon.a (mon.0) 205 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:01.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:00 smithi204 bash[35809]: cluster 2024-02-10T09:18:59.573586+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e9: 1 total, 0 up, 0 in 2024-02-10T09:19:01.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:00 smithi204 bash[35809]: audit 2024-02-10T09:18:59.574057+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:01.908 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: cluster 2024-02-10T09:19:00.579345+0000 mon.a (mon.0) 208 : cluster [INF] osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362] boot 2024-02-10T09:19:01.908 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: cluster 2024-02-10T09:19:00.579374+0000 mon.a (mon.0) 209 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: audit 2024-02-10T09:19:00.579878+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: cluster 2024-02-10T09:19:00.599270+0000 mgr.y (mgr.14140) 60 : cluster [DBG] pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: audit 2024-02-10T09:19:01.161234+0000 mon.a (mon.0) 211 : audit [INF] from='client.? 172.21.15.153:0/993063970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa508d13-ae9b-4d62-b567-1be1a9d46d5c"}]: dispatch 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: audit 2024-02-10T09:19:01.165721+0000 mon.a (mon.0) 212 : audit [INF] from='client.? 172.21.15.153:0/993063970' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa508d13-ae9b-4d62-b567-1be1a9d46d5c"}]': finished 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: cluster 2024-02-10T09:19:01.165779+0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e11: 2 total, 1 up, 1 in 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[39117]: audit 2024-02-10T09:19:01.165895+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: cluster 2024-02-10T09:19:00.579345+0000 mon.a (mon.0) 208 : cluster [INF] osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362] boot 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: cluster 2024-02-10T09:19:00.579374+0000 mon.a (mon.0) 209 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: audit 2024-02-10T09:19:00.579878+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: cluster 2024-02-10T09:19:00.599270+0000 mgr.y (mgr.14140) 60 : cluster [DBG] pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: audit 2024-02-10T09:19:01.161234+0000 mon.a (mon.0) 211 : audit [INF] from='client.? 172.21.15.153:0/993063970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa508d13-ae9b-4d62-b567-1be1a9d46d5c"}]: dispatch 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: audit 2024-02-10T09:19:01.165721+0000 mon.a (mon.0) 212 : audit [INF] from='client.? 172.21.15.153:0/993063970' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa508d13-ae9b-4d62-b567-1be1a9d46d5c"}]': finished 2024-02-10T09:19:01.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: cluster 2024-02-10T09:19:01.165779+0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e11: 2 total, 1 up, 1 in 2024-02-10T09:19:01.910 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:01 smithi153 bash[35780]: audit 2024-02-10T09:19:01.165895+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: cluster 2024-02-10T09:19:00.579345+0000 mon.a (mon.0) 208 : cluster [INF] osd.0 [v2:172.21.15.153:6802/2353941362,v1:172.21.15.153:6803/2353941362] boot 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: cluster 2024-02-10T09:19:00.579374+0000 mon.a (mon.0) 209 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: audit 2024-02-10T09:19:00.579878+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: cluster 2024-02-10T09:19:00.599270+0000 mgr.y (mgr.14140) 60 : cluster [DBG] pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: audit 2024-02-10T09:19:01.161234+0000 mon.a (mon.0) 211 : audit [INF] from='client.? 172.21.15.153:0/993063970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa508d13-ae9b-4d62-b567-1be1a9d46d5c"}]: dispatch 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: audit 2024-02-10T09:19:01.165721+0000 mon.a (mon.0) 212 : audit [INF] from='client.? 172.21.15.153:0/993063970' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa508d13-ae9b-4d62-b567-1be1a9d46d5c"}]': finished 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: cluster 2024-02-10T09:19:01.165779+0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e11: 2 total, 1 up, 1 in 2024-02-10T09:19:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:01 smithi204 bash[35809]: audit 2024-02-10T09:19:01.165895+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[35780]: audit 2024-02-10T09:19:01.942667+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.153:0/77511032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[35780]: cluster 2024-02-10T09:19:02.166805+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e12: 2 total, 1 up, 1 in 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[35780]: audit 2024-02-10T09:19:02.166910+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[35780]: audit 2024-02-10T09:19:02.544301+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[39117]: audit 2024-02-10T09:19:01.942667+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.153:0/77511032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[39117]: cluster 2024-02-10T09:19:02.166805+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e12: 2 total, 1 up, 1 in 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[39117]: audit 2024-02-10T09:19:02.166910+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:02 smithi153 bash[39117]: audit 2024-02-10T09:19:02.544301+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:02 smithi204 bash[35809]: audit 2024-02-10T09:19:01.942667+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.153:0/77511032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:02 smithi204 bash[35809]: cluster 2024-02-10T09:19:02.166805+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e12: 2 total, 1 up, 1 in 2024-02-10T09:19:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:02 smithi204 bash[35809]: audit 2024-02-10T09:19:02.166910+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:02 smithi204 bash[35809]: audit 2024-02-10T09:19:02.544301+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:03.856 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:03 smithi153 bash[35780]: cluster 2024-02-10T09:19:02.599528+0000 mgr.y (mgr.14140) 61 : cluster [DBG] pgmap v51: 1 pgs: 1 undersized+peered; 0 B data, 96 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:03.857 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:03 smithi153 bash[39117]: cluster 2024-02-10T09:19:02.599528+0000 mgr.y (mgr.14140) 61 : cluster [DBG] pgmap v51: 1 pgs: 1 undersized+peered; 0 B data, 96 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:03.857 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:03 smithi153 bash[35780]: audit 2024-02-10T09:19:02.726919+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:03.857 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:03 smithi153 bash[35780]: audit 2024-02-10T09:19:02.730268+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:03.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:03 smithi153 bash[39117]: audit 2024-02-10T09:19:02.726919+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:03.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:03 smithi153 bash[39117]: audit 2024-02-10T09:19:02.730268+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:03 smithi204 bash[35809]: cluster 2024-02-10T09:19:02.599528+0000 mgr.y (mgr.14140) 61 : cluster [DBG] pgmap v51: 1 pgs: 1 undersized+peered; 0 B data, 96 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:03 smithi204 bash[35809]: audit 2024-02-10T09:19:02.726919+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:03 smithi204 bash[35809]: audit 2024-02-10T09:19:02.730268+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"056929e39805\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958204\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:18:55.643245\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:50.609394\"}}, \"last_daemon_update\": \"2024-02-10T09:18:56.958269\", \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:04.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:04 smithi153 bash[35780]: audit 2024-02-10T09:19:04.434280+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:04.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:04 smithi153 bash[39117]: audit 2024-02-10T09:19:04.434280+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:05.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:04 smithi204 bash[35809]: audit 2024-02-10T09:19:04.434280+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:05.819 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:05 smithi153 bash[35780]: cluster 2024-02-10T09:19:04.599788+0000 mgr.y (mgr.14140) 62 : cluster [DBG] pgmap v52: 1 pgs: 1 undersized+peered; 0 B data, 96 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:05.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:05 smithi153 bash[39117]: cluster 2024-02-10T09:19:04.599788+0000 mgr.y (mgr.14140) 62 : cluster [DBG] pgmap v52: 1 pgs: 1 undersized+peered; 0 B data, 96 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:06.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:05 smithi204 bash[35809]: cluster 2024-02-10T09:19:04.599788+0000 mgr.y (mgr.14140) 62 : cluster [DBG] pgmap v52: 1 pgs: 1 undersized+peered; 0 B data, 96 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:07.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:07 smithi204 bash[35809]: cluster 2024-02-10T09:19:06.600047+0000 mgr.y (mgr.14140) 63 : cluster [DBG] pgmap v53: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:07.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:07 smithi204 bash[35809]: audit 2024-02-10T09:19:07.111959+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:19:07.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:07 smithi204 bash[35809]: audit 2024-02-10T09:19:07.112477+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:07.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:07 smithi204 bash[35809]: audit 2024-02-10T09:19:07.113044+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[35780]: cluster 2024-02-10T09:19:06.600047+0000 mgr.y (mgr.14140) 63 : cluster [DBG] pgmap v53: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[35780]: audit 2024-02-10T09:19:07.111959+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[35780]: audit 2024-02-10T09:19:07.112477+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[35780]: audit 2024-02-10T09:19:07.113044+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[39117]: cluster 2024-02-10T09:19:06.600047+0000 mgr.y (mgr.14140) 63 : cluster [DBG] pgmap v53: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[39117]: audit 2024-02-10T09:19:07.111959+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[39117]: audit 2024-02-10T09:19:07.112477+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:07 smithi153 bash[39117]: audit 2024-02-10T09:19:07.113044+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2024-02-10T09:19:08.708 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:08 smithi153 bash[39117]: cephadm 2024-02-10T09:19:07.112858+0000 mgr.y (mgr.14140) 64 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:19:08.708 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:08 smithi153 bash[35780]: cephadm 2024-02-10T09:19:07.112858+0000 mgr.y (mgr.14140) 64 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:19:08.708 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:08 smithi153 systemd[1]: Stopping Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:19:08.708 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:08 smithi153 bash[41523]: debug 2024-02-10T09:19:08.705+0000 7f5f5e877700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:19:08.709 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:08 smithi153 bash[41523]: debug 2024-02-10T09:19:08.705+0000 7f5f5e877700 -1 osd.0 12 *** Got signal Terminated *** 2024-02-10T09:19:08.709 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:08 smithi153 bash[41523]: debug 2024-02-10T09:19:08.705+0000 7f5f5e877700 -1 osd.0 12 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:19:08.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:08 smithi204 bash[35809]: cephadm 2024-02-10T09:19:07.112858+0000 mgr.y (mgr.14140) 64 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:19:09.358 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:09 smithi153 podman[42669]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.0 2024-02-10T09:19:09.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:09 smithi204 bash[35809]: cluster 2024-02-10T09:19:08.600506+0000 mgr.y (mgr.14140) 65 : cluster [DBG] pgmap v54: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:09.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:09 smithi153 bash[39117]: cluster 2024-02-10T09:19:08.600506+0000 mgr.y (mgr.14140) 65 : cluster [DBG] pgmap v54: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:09.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:09 smithi153 bash[35780]: cluster 2024-02-10T09:19:08.600506+0000 mgr.y (mgr.14140) 65 : cluster [DBG] pgmap v54: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:10.872 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:10 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.0.service: Succeeded. 2024-02-10T09:19:10.872 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:10 smithi153 systemd[1]: Stopped Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:19:10.872 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:10 smithi153 systemd[1]: Starting Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:19:10.872 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:10 smithi153 podman[42804]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T09:19:10.872 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:10 smithi153 systemd[1]: Started Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[35780]: audit 2024-02-10T09:19:10.635816+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[35780]: audit 2024-02-10T09:19:10.636186+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[35780]: audit 2024-02-10T09:19:10.638889+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[35780]: audit 2024-02-10T09:19:10.639746+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[35780]: audit 2024-02-10T09:19:10.640772+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[35780]: audit 2024-02-10T09:19:10.641397+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2024-02-10T09:19:10.873 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[39117]: audit 2024-02-10T09:19:10.635816+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:10.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[39117]: audit 2024-02-10T09:19:10.636186+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:10.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[39117]: audit 2024-02-10T09:19:10.638889+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:10.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[39117]: audit 2024-02-10T09:19:10.639746+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:19:10.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[39117]: audit 2024-02-10T09:19:10.640772+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:10.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:10 smithi153 bash[39117]: audit 2024-02-10T09:19:10.641397+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2024-02-10T09:19:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:10 smithi204 bash[35809]: audit 2024-02-10T09:19:10.635816+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:10 smithi204 bash[35809]: audit 2024-02-10T09:19:10.636186+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:10 smithi204 bash[35809]: audit 2024-02-10T09:19:10.638889+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:10 smithi204 bash[35809]: audit 2024-02-10T09:19:10.639746+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:19:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:10 smithi204 bash[35809]: audit 2024-02-10T09:19:10.640772+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:10 smithi204 bash[35809]: audit 2024-02-10T09:19:10.641397+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2024-02-10T09:19:11.688 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T09:19:11.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-02-10T09:19:11.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T09:19:11.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-02-10T09:19:11.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T09:19:11.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T09:19:11.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[42822]: --> ceph-volume lvm activate successful for osd ID: 0 2024-02-10T09:19:12.006 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[35780]: cluster 2024-02-10T09:19:10.600837+0000 mgr.y (mgr.14140) 66 : cluster [DBG] pgmap v55: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:12.006 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[35780]: cephadm 2024-02-10T09:19:10.641205+0000 mgr.y (mgr.14140) 67 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:19:12.006 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[39117]: cluster 2024-02-10T09:19:10.600837+0000 mgr.y (mgr.14140) 66 : cluster [DBG] pgmap v55: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:12.006 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:11 smithi153 bash[39117]: cephadm 2024-02-10T09:19:10.641205+0000 mgr.y (mgr.14140) 67 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:19:12.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:11 smithi204 bash[35809]: cluster 2024-02-10T09:19:10.600837+0000 mgr.y (mgr.14140) 66 : cluster [DBG] pgmap v55: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:12.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:11 smithi204 bash[35809]: cephadm 2024-02-10T09:19:10.641205+0000 mgr.y (mgr.14140) 67 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:19:12.965 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 0,1 on host 'smithi153' 2024-02-10T09:19:13.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:13 smithi153 bash[35780]: cluster 2024-02-10T09:19:12.601135+0000 mgr.y (mgr.14140) 68 : cluster [DBG] pgmap v56: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:13.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:13 smithi153 bash[39117]: cluster 2024-02-10T09:19:12.601135+0000 mgr.y (mgr.14140) 68 : cluster [DBG] pgmap v56: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:13.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:13 smithi153 bash[35780]: audit 2024-02-10T09:19:12.959512+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:13.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:13 smithi153 bash[35780]: audit 2024-02-10T09:19:12.963443+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:13.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:13 smithi153 bash[39117]: audit 2024-02-10T09:19:12.959512+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:13.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:13 smithi153 bash[39117]: audit 2024-02-10T09:19:12.963443+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:13 smithi204 bash[35809]: cluster 2024-02-10T09:19:12.601135+0000 mgr.y (mgr.14140) 68 : cluster [DBG] pgmap v56: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:13 smithi204 bash[35809]: audit 2024-02-10T09:19:12.959512+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:13 smithi204 bash[35809]: audit 2024-02-10T09:19:12.963443+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958055\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958146\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:56.958177\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_device_update\": \"2024-02-10T09:19:02.726251\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:14.235 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T09:19:14.238 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_2... 2024-02-10T09:19:14.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T09:19:14.269 INFO:journalctl@ceph.osd.1.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:19:15.762 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:15 smithi153 bash[35780]: cluster 2024-02-10T09:19:14.601632+0000 mgr.y (mgr.14140) 69 : cluster [DBG] pgmap v57: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:15.762 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:15 smithi153 bash[39117]: cluster 2024-02-10T09:19:14.601632+0000 mgr.y (mgr.14140) 69 : cluster [DBG] pgmap v57: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:16.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:15 smithi204 bash[35809]: cluster 2024-02-10T09:19:14.601632+0000 mgr.y (mgr.14140) 69 : cluster [DBG] pgmap v57: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:16.346 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T09:19:16.346 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T09:19:16.346 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-02-10T09:19:16.346 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-02-10T09:19:16.346 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.015129 s, 693 MB/s 2024-02-10T09:19:16.346 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-02-10T09:19:16.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T09:19:16.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-02-10T09:19:16.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T09:19:16.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-02-10T09:19:16.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T09:19:16.391 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T09:19:16.391 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43103]: --> ceph-volume lvm activate successful for osd ID: 1 2024-02-10T09:19:16.688 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43214]: debug 2024-02-10T09:19:15.277+0000 7f316f914ec0 -1 Falling back to public interface 2024-02-10T09:19:16.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43214]: debug 2024-02-10T09:19:15.827+0000 7f316f914ec0 -1 osd.0 12 log_to_monitors {default=true} 2024-02-10T09:19:16.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[43214]: debug 2024-02-10T09:19:15.835+0000 7f315f8dd700 -1 osd.0 12 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:19:17.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:15.832153+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:19:17.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:15.832444+0000 mon.a (mon.0) 232 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:19:17.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:15.832871+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:17.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:15.833077+0000 mon.a (mon.0) 233 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:17.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:16.494993+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:17.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:16.497829+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:17.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:16 smithi204 bash[35809]: audit 2024-02-10T09:19:16.498350+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:17.109 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:15.832153+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:19:17.109 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:15.832444+0000 mon.a (mon.0) 232 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:19:17.109 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:15.832871+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:15.833077+0000 mon.a (mon.0) 233 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:16.494993+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:16.497829+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[35780]: audit 2024-02-10T09:19:16.498350+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:15.832153+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:15.832444+0000 mon.a (mon.0) 232 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:19:17.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:15.832871+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:17.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:15.833077+0000 mon.a (mon.0) 233 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:17.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:16.494993+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:17.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:16.497829+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:17.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:16 smithi153 bash[39117]: audit 2024-02-10T09:19:16.498350+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:17.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-02-10T09:19:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:17 smithi204 bash[35809]: cluster 2024-02-10T09:19:16.601925+0000 mgr.y (mgr.14140) 70 : cluster [DBG] pgmap v58: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:17 smithi204 bash[35809]: cluster 2024-02-10T09:19:16.704671+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:19:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:17 smithi204 bash[35809]: cluster 2024-02-10T09:19:16.704695+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN) 2024-02-10T09:19:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:17 smithi204 bash[35809]: cluster 2024-02-10T09:19:16.704715+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 root (1 osds) down (OSD_ROOT_DOWN) 2024-02-10T09:19:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:17 smithi204 bash[35809]: cluster 2024-02-10T09:19:16.714337+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e13: 2 total, 0 up, 1 in 2024-02-10T09:19:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:17 smithi204 bash[35809]: audit 2024-02-10T09:19:16.714433+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[35780]: cluster 2024-02-10T09:19:16.601925+0000 mgr.y (mgr.14140) 70 : cluster [DBG] pgmap v58: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[35780]: cluster 2024-02-10T09:19:16.704671+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[35780]: cluster 2024-02-10T09:19:16.704695+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN) 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[35780]: cluster 2024-02-10T09:19:16.704715+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 root (1 osds) down (OSD_ROOT_DOWN) 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[35780]: cluster 2024-02-10T09:19:16.714337+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e13: 2 total, 0 up, 1 in 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[35780]: audit 2024-02-10T09:19:16.714433+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[39117]: cluster 2024-02-10T09:19:16.601925+0000 mgr.y (mgr.14140) 70 : cluster [DBG] pgmap v58: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[39117]: cluster 2024-02-10T09:19:16.704671+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[39117]: cluster 2024-02-10T09:19:16.704695+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN) 2024-02-10T09:19:18.126 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[39117]: cluster 2024-02-10T09:19:16.704715+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 root (1 osds) down (OSD_ROOT_DOWN) 2024-02-10T09:19:18.127 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[39117]: cluster 2024-02-10T09:19:16.714337+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e13: 2 total, 0 up, 1 in 2024-02-10T09:19:18.127 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:17 smithi153 bash[39117]: audit 2024-02-10T09:19:16.714433+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:18.992 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: cluster 2024-02-10T09:19:17.711139+0000 mon.a (mon.0) 242 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:19:18.992 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: cluster 2024-02-10T09:19:17.711161+0000 mon.a (mon.0) 243 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (1 osds) down) 2024-02-10T09:19:18.992 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: cluster 2024-02-10T09:19:17.711174+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: OSD_ROOT_DOWN (was: 1 root (1 osds) down) 2024-02-10T09:19:18.992 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: cluster 2024-02-10T09:19:17.717937+0000 mon.a (mon.0) 245 : cluster [INF] osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228] boot 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: cluster 2024-02-10T09:19:17.717977+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e14: 2 total, 1 up, 1 in 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: audit 2024-02-10T09:19:17.718125+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[39117]: audit 2024-02-10T09:19:17.718467+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: cluster 2024-02-10T09:19:17.711139+0000 mon.a (mon.0) 242 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: cluster 2024-02-10T09:19:17.711161+0000 mon.a (mon.0) 243 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (1 osds) down) 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: cluster 2024-02-10T09:19:17.711174+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: OSD_ROOT_DOWN (was: 1 root (1 osds) down) 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: cluster 2024-02-10T09:19:17.717937+0000 mon.a (mon.0) 245 : cluster [INF] osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228] boot 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: cluster 2024-02-10T09:19:17.717977+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e14: 2 total, 1 up, 1 in 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: audit 2024-02-10T09:19:17.718125+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:18.993 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:18 smithi153 bash[35780]: audit 2024-02-10T09:19:17.718467+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: cluster 2024-02-10T09:19:17.711139+0000 mon.a (mon.0) 242 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: cluster 2024-02-10T09:19:17.711161+0000 mon.a (mon.0) 243 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (1 osds) down) 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: cluster 2024-02-10T09:19:17.711174+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: OSD_ROOT_DOWN (was: 1 root (1 osds) down) 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: cluster 2024-02-10T09:19:17.717937+0000 mon.a (mon.0) 245 : cluster [INF] osd.0 [v2:172.21.15.153:6802/2774397228,v1:172.21.15.153:6803/2774397228] boot 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: cluster 2024-02-10T09:19:17.717977+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e14: 2 total, 1 up, 1 in 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: audit 2024-02-10T09:19:17.718125+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:19:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:18 smithi204 bash[35809]: audit 2024-02-10T09:19:17.718467+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:19.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:19 smithi153 bash[39117]: cluster 2024-02-10T09:19:18.602194+0000 mgr.y (mgr.14140) 71 : cluster [DBG] pgmap v61: 1 pgs: 1 stale+undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:19.901 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:19 smithi153 bash[39117]: cluster 2024-02-10T09:19:18.724293+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e15: 2 total, 1 up, 1 in 2024-02-10T09:19:19.901 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:19 smithi153 bash[39117]: audit 2024-02-10T09:19:18.724422+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:19.901 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:19 smithi153 bash[35780]: cluster 2024-02-10T09:19:18.602194+0000 mgr.y (mgr.14140) 71 : cluster [DBG] pgmap v61: 1 pgs: 1 stale+undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:19.901 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:19 smithi153 bash[35780]: cluster 2024-02-10T09:19:18.724293+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e15: 2 total, 1 up, 1 in 2024-02-10T09:19:19.901 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:19 smithi153 bash[35780]: audit 2024-02-10T09:19:18.724422+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:20.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:19 smithi204 bash[35809]: cluster 2024-02-10T09:19:18.602194+0000 mgr.y (mgr.14140) 71 : cluster [DBG] pgmap v61: 1 pgs: 1 stale+undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:20.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:19 smithi204 bash[35809]: cluster 2024-02-10T09:19:18.724293+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e15: 2 total, 1 up, 1 in 2024-02-10T09:19:20.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:19 smithi204 bash[35809]: audit 2024-02-10T09:19:18.724422+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:20.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:20 smithi153 bash[43749]: debug 2024-02-10T09:19:20.549+0000 7f553b996ec0 -1 Falling back to public interface 2024-02-10T09:19:21.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[43749]: debug 2024-02-10T09:19:21.104+0000 7f553b996ec0 -1 osd.1 0 log_to_monitors {default=true} 2024-02-10T09:19:22.033 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: cluster 2024-02-10T09:19:20.602529+0000 mgr.y (mgr.14140) 72 : cluster [DBG] pgmap v63: 1 pgs: 1 stale+undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:22.033 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: audit 2024-02-10T09:19:21.107268+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:22.033 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: audit 2024-02-10T09:19:21.107511+0000 mon.a (mon.0) 251 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: audit 2024-02-10T09:19:21.614360+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: audit 2024-02-10T09:19:21.626032+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: audit 2024-02-10T09:19:21.626559+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[35780]: audit 2024-02-10T09:19:21.627246+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: cluster 2024-02-10T09:19:20.602529+0000 mgr.y (mgr.14140) 72 : cluster [DBG] pgmap v63: 1 pgs: 1 stale+undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: audit 2024-02-10T09:19:21.107268+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: audit 2024-02-10T09:19:21.107511+0000 mon.a (mon.0) 251 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: audit 2024-02-10T09:19:21.614360+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: audit 2024-02-10T09:19:21.626032+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: audit 2024-02-10T09:19:21.626559+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:21 smithi153 bash[39117]: audit 2024-02-10T09:19:21.627246+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: cluster 2024-02-10T09:19:20.602529+0000 mgr.y (mgr.14140) 72 : cluster [DBG] pgmap v63: 1 pgs: 1 stale+undersized+peered; 0 B data, 112 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: audit 2024-02-10T09:19:21.107268+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: audit 2024-02-10T09:19:21.107511+0000 mon.a (mon.0) 251 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: audit 2024-02-10T09:19:21.614360+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: audit 2024-02-10T09:19:21.626032+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: audit 2024-02-10T09:19:21.626559+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:21 smithi204 bash[35809]: audit 2024-02-10T09:19:21.627246+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.625264+0000 mgr.y (mgr.14140) 73 : audit [DBG] from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:19:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: cephadm 2024-02-10T09:19:21.625807+0000 mgr.y (mgr.14140) 74 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:19:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.735668+0000 mon.a (mon.0) 256 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: cluster 2024-02-10T09:19:21.735719+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e16: 2 total, 1 up, 1 in 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.736183+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.736220+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.736605+0000 mon.a (mon.0) 259 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.805950+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.809486+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:21.811223+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:22.732016+0000 mon.a (mon.0) 263 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: cluster 2024-02-10T09:19:22.732050+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e17: 2 total, 1 up, 1 in 2024-02-10T09:19:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:22 smithi204 bash[35809]: audit 2024-02-10T09:19:22.732178+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:23.121 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[43749]: debug 2024-02-10T09:19:22.733+0000 7f5520743700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T09:19:23.121 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[43749]: debug 2024-02-10T09:19:22.736+0000 7f552b95f700 -1 osd.1 17 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.625264+0000 mgr.y (mgr.14140) 73 : audit [DBG] from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: cephadm 2024-02-10T09:19:21.625807+0000 mgr.y (mgr.14140) 74 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.735668+0000 mon.a (mon.0) 256 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: cluster 2024-02-10T09:19:21.735719+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e16: 2 total, 1 up, 1 in 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.736183+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.736220+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:23.121 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.736605+0000 mon.a (mon.0) 259 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.625264+0000 mgr.y (mgr.14140) 73 : audit [DBG] from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: cephadm 2024-02-10T09:19:21.625807+0000 mgr.y (mgr.14140) 74 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.735668+0000 mon.a (mon.0) 256 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: cluster 2024-02-10T09:19:21.735719+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e16: 2 total, 1 up, 1 in 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.736183+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.736220+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.736605+0000 mon.a (mon.0) 259 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:23.122 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.805950+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.805950+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.809486+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:21.811223+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:22.732016+0000 mon.a (mon.0) 263 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: cluster 2024-02-10T09:19:22.732050+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e17: 2 total, 1 up, 1 in 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[35780]: audit 2024-02-10T09:19:22.732178+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:23.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.809486+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494189\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494264\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494297\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:16.494326\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:16.494394\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:23.124 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:21.811223+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:23.124 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:22.732016+0000 mon.a (mon.0) 263 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:23.124 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: cluster 2024-02-10T09:19:22.732050+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e17: 2 total, 1 up, 1 in 2024-02-10T09:19:23.124 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:22 smithi153 bash[39117]: audit 2024-02-10T09:19:22.732178+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:23 smithi204 bash[35809]: cluster 2024-02-10T09:19:22.123305+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:19:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:23 smithi204 bash[35809]: cluster 2024-02-10T09:19:22.123351+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:23 smithi204 bash[35809]: cluster 2024-02-10T09:19:22.602830+0000 mgr.y (mgr.14140) 75 : cluster [DBG] pgmap v65: 1 pgs: 1 undersized+peered; 0 B data, 136 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:24.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:23 smithi153 bash[39117]: cluster 2024-02-10T09:19:22.123305+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:19:24.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:23 smithi153 bash[39117]: cluster 2024-02-10T09:19:22.123351+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:24.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:23 smithi153 bash[39117]: cluster 2024-02-10T09:19:22.602830+0000 mgr.y (mgr.14140) 75 : cluster [DBG] pgmap v65: 1 pgs: 1 undersized+peered; 0 B data, 136 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:23 smithi153 bash[35780]: cluster 2024-02-10T09:19:22.123305+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:19:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:23 smithi153 bash[35780]: cluster 2024-02-10T09:19:22.123351+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:23 smithi153 bash[35780]: cluster 2024-02-10T09:19:22.602830+0000 mgr.y (mgr.14140) 75 : cluster [DBG] pgmap v65: 1 pgs: 1 undersized+peered; 0 B data, 136 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: cluster 2024-02-10T09:19:23.748623+0000 mon.a (mon.0) 266 : cluster [INF] osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523] boot 2024-02-10T09:19:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: cluster 2024-02-10T09:19:23.748664+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e18: 2 total, 2 up, 2 in 2024-02-10T09:19:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: audit 2024-02-10T09:19:23.749226+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:25.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: audit 2024-02-10T09:19:24.077911+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"5bd99e68e07b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077222\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:18.635344\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:24.077287\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:25.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: audit 2024-02-10T09:19:24.081271+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"5bd99e68e07b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077222\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:18.635344\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:24.077287\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:25.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: audit 2024-02-10T09:19:24.402896+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.153:0/1711300254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e5a960-d7e8-4e2c-b913-6848f7f0de36"}]: dispatch 2024-02-10T09:19:25.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: audit 2024-02-10T09:19:24.408955+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.153:0/1711300254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6e5a960-d7e8-4e2c-b913-6848f7f0de36"}]': finished 2024-02-10T09:19:25.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: cluster 2024-02-10T09:19:24.409001+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e19: 3 total, 2 up, 2 in 2024-02-10T09:19:25.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:24 smithi204 bash[35809]: audit 2024-02-10T09:19:24.409128+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: cluster 2024-02-10T09:19:23.748623+0000 mon.a (mon.0) 266 : cluster [INF] osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523] boot 2024-02-10T09:19:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: cluster 2024-02-10T09:19:23.748664+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e18: 2 total, 2 up, 2 in 2024-02-10T09:19:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: audit 2024-02-10T09:19:23.749226+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: cluster 2024-02-10T09:19:23.748623+0000 mon.a (mon.0) 266 : cluster [INF] osd.1 [v2:172.21.15.153:6810/160758523,v1:172.21.15.153:6811/160758523] boot 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: cluster 2024-02-10T09:19:23.748664+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e18: 2 total, 2 up, 2 in 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: audit 2024-02-10T09:19:23.749226+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: audit 2024-02-10T09:19:24.077911+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"5bd99e68e07b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077222\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:18.635344\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:24.077287\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: audit 2024-02-10T09:19:24.081271+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"5bd99e68e07b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077222\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:18.635344\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:24.077287\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: audit 2024-02-10T09:19:24.402896+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.153:0/1711300254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e5a960-d7e8-4e2c-b913-6848f7f0de36"}]: dispatch 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: audit 2024-02-10T09:19:24.408955+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.153:0/1711300254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6e5a960-d7e8-4e2c-b913-6848f7f0de36"}]': finished 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: cluster 2024-02-10T09:19:24.409001+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e19: 3 total, 2 up, 2 in 2024-02-10T09:19:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[35780]: audit 2024-02-10T09:19:24.409128+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: audit 2024-02-10T09:19:24.077911+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"5bd99e68e07b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077222\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:18.635344\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:24.077287\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: audit 2024-02-10T09:19:24.081271+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"5bd99e68e07b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077222\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:18.635344\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:10.639514\"}}, \"last_daemon_update\": \"2024-02-10T09:19:24.077287\", \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: audit 2024-02-10T09:19:24.402896+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.153:0/1711300254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e5a960-d7e8-4e2c-b913-6848f7f0de36"}]: dispatch 2024-02-10T09:19:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: audit 2024-02-10T09:19:24.408955+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.153:0/1711300254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6e5a960-d7e8-4e2c-b913-6848f7f0de36"}]': finished 2024-02-10T09:19:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: cluster 2024-02-10T09:19:24.409001+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e19: 3 total, 2 up, 2 in 2024-02-10T09:19:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:24 smithi153 bash[39117]: audit 2024-02-10T09:19:24.409128+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:25 smithi204 bash[35809]: cluster 2024-02-10T09:19:24.603157+0000 mgr.y (mgr.14140) 76 : cluster [DBG] pgmap v69: 1 pgs: 1 undersized+peered; 0 B data, 136 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:25 smithi204 bash[35809]: audit 2024-02-10T09:19:25.173025+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.153:0/2090221622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:25 smithi204 bash[35809]: cluster 2024-02-10T09:19:25.411679+0000 mon.a (mon.0) 275 : cluster [DBG] osdmap e20: 3 total, 2 up, 2 in 2024-02-10T09:19:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:25 smithi204 bash[35809]: audit 2024-02-10T09:19:25.411813+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:25 smithi204 bash[35809]: audit 2024-02-10T09:19:25.414530+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/devicehealth/last_scrape","val":"20240210-091720"}]: dispatch 2024-02-10T09:19:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:25 smithi204 bash[35809]: audit 2024-02-10T09:19:25.419687+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/devicehealth/last_scrape","val":"20240210-091720"}]': finished 2024-02-10T09:19:26.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[35780]: cluster 2024-02-10T09:19:24.603157+0000 mgr.y (mgr.14140) 76 : cluster [DBG] pgmap v69: 1 pgs: 1 undersized+peered; 0 B data, 136 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:26.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[35780]: audit 2024-02-10T09:19:25.173025+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.153:0/2090221622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[35780]: cluster 2024-02-10T09:19:25.411679+0000 mon.a (mon.0) 275 : cluster [DBG] osdmap e20: 3 total, 2 up, 2 in 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[35780]: audit 2024-02-10T09:19:25.411813+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[35780]: audit 2024-02-10T09:19:25.414530+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/devicehealth/last_scrape","val":"20240210-091720"}]: dispatch 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[35780]: audit 2024-02-10T09:19:25.419687+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/devicehealth/last_scrape","val":"20240210-091720"}]': finished 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[39117]: cluster 2024-02-10T09:19:24.603157+0000 mgr.y (mgr.14140) 76 : cluster [DBG] pgmap v69: 1 pgs: 1 undersized+peered; 0 B data, 136 KiB used, 88 GiB / 89 GiB avail 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[39117]: audit 2024-02-10T09:19:25.173025+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.153:0/2090221622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[39117]: cluster 2024-02-10T09:19:25.411679+0000 mon.a (mon.0) 275 : cluster [DBG] osdmap e20: 3 total, 2 up, 2 in 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[39117]: audit 2024-02-10T09:19:25.411813+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[39117]: audit 2024-02-10T09:19:25.414530+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/devicehealth/last_scrape","val":"20240210-091720"}]: dispatch 2024-02-10T09:19:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:25 smithi153 bash[39117]: audit 2024-02-10T09:19:25.419687+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/devicehealth/last_scrape","val":"20240210-091720"}]': finished 2024-02-10T09:19:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:27 smithi153 bash[35780]: cluster 2024-02-10T09:19:26.603635+0000 mgr.y (mgr.14140) 77 : cluster [DBG] pgmap v71: 1 pgs: 1 undersized+peered; 0 B data, 328 KiB used, 177 GiB / 179 GiB avail 2024-02-10T09:19:27.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:27 smithi153 bash[39117]: cluster 2024-02-10T09:19:26.603635+0000 mgr.y (mgr.14140) 77 : cluster [DBG] pgmap v71: 1 pgs: 1 undersized+peered; 0 B data, 328 KiB used, 177 GiB / 179 GiB avail 2024-02-10T09:19:27.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:27 smithi204 bash[35809]: cluster 2024-02-10T09:19:26.603635+0000 mgr.y (mgr.14140) 77 : cluster [DBG] pgmap v71: 1 pgs: 1 undersized+peered; 0 B data, 328 KiB used, 177 GiB / 179 GiB avail 2024-02-10T09:19:28.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:28 smithi204 bash[35809]: audit 2024-02-10T09:19:27.612618+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:28.879 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:28 smithi153 bash[35780]: audit 2024-02-10T09:19:27.612618+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:28.879 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:28 smithi153 bash[39117]: audit 2024-02-10T09:19:27.612618+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:29.846 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:29 smithi153 bash[35780]: cluster 2024-02-10T09:19:28.604063+0000 mgr.y (mgr.14140) 78 : cluster [DBG] pgmap v72: 1 pgs: 1 undersized+peered; 0 B data, 328 KiB used, 177 GiB / 179 GiB avail 2024-02-10T09:19:29.846 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:29 smithi153 bash[39117]: cluster 2024-02-10T09:19:28.604063+0000 mgr.y (mgr.14140) 78 : cluster [DBG] pgmap v72: 1 pgs: 1 undersized+peered; 0 B data, 328 KiB used, 177 GiB / 179 GiB avail 2024-02-10T09:19:30.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:29 smithi204 bash[35809]: cluster 2024-02-10T09:19:28.604063+0000 mgr.y (mgr.14140) 78 : cluster [DBG] pgmap v72: 1 pgs: 1 undersized+peered; 0 B data, 328 KiB used, 177 GiB / 179 GiB avail 2024-02-10T09:19:30.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:30 smithi153 bash[35780]: audit 2024-02-10T09:19:30.146338+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:19:30.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:30 smithi153 bash[35780]: audit 2024-02-10T09:19:30.146861+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:30.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:30 smithi153 bash[35780]: audit 2024-02-10T09:19:30.147379+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2024-02-10T09:19:30.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:30 smithi153 bash[39117]: audit 2024-02-10T09:19:30.146338+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:19:30.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:30 smithi153 bash[39117]: audit 2024-02-10T09:19:30.146861+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:30.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:30 smithi153 bash[39117]: audit 2024-02-10T09:19:30.147379+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2024-02-10T09:19:31.080 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:30 smithi204 bash[35809]: audit 2024-02-10T09:19:30.146338+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:19:31.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:30 smithi204 bash[35809]: audit 2024-02-10T09:19:30.146861+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:31.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:30 smithi204 bash[35809]: audit 2024-02-10T09:19:30.147379+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2024-02-10T09:19:31.595 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:31 smithi153 systemd[1]: Stopping Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:19:31.595 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[43749]: debug 2024-02-10T09:19:31.540+0000 7f55331de700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:19:31.595 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[43749]: debug 2024-02-10T09:19:31.540+0000 7f55331de700 -1 osd.1 20 *** Got signal Terminated *** 2024-02-10T09:19:31.595 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[43749]: debug 2024-02-10T09:19:31.540+0000 7f55331de700 -1 osd.1 20 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:19:31.595 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[35780]: cephadm 2024-02-10T09:19:30.147215+0000 mgr.y (mgr.14140) 79 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:19:31.596 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[35780]: cluster 2024-02-10T09:19:30.604328+0000 mgr.y (mgr.14140) 80 : cluster [DBG] pgmap v73: 1 pgs: 1 active+undersized; 0 B data, 340 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[39117]: cephadm 2024-02-10T09:19:30.147215+0000 mgr.y (mgr.14140) 79 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:19:31.914 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:31 smithi153 bash[39117]: cluster 2024-02-10T09:19:30.604328+0000 mgr.y (mgr.14140) 80 : cluster [DBG] pgmap v73: 1 pgs: 1 active+undersized; 0 B data, 340 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:31 smithi204 bash[35809]: cephadm 2024-02-10T09:19:30.147215+0000 mgr.y (mgr.14140) 79 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:19:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:31 smithi204 bash[35809]: cluster 2024-02-10T09:19:30.604328+0000 mgr.y (mgr.14140) 80 : cluster [DBG] pgmap v73: 1 pgs: 1 active+undersized; 0 B data, 340 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:32.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:32 smithi153 podman[44878]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.1 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:31.586296+0000 mon.a (mon.0) 283 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:31.586316+0000 mon.a (mon.0) 284 : cluster [INF] Cluster is now healthy 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:31.795268+0000 mon.a (mon.0) 285 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:31.795298+0000 mon.a (mon.0) 286 : cluster [INF] osd.1 failed (root=default,host=smithi153) (connection refused reported by osd.0) 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:31.795460+0000 mon.a (mon.0) 287 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:32.196278+0000 mon.a (mon.0) 288 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[39117]: cluster 2024-02-10T09:19:32.196377+0000 mon.a (mon.0) 289 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.833 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:31.586296+0000 mon.a (mon.0) 283 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-10T09:19:32.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:31.586316+0000 mon.a (mon.0) 284 : cluster [INF] Cluster is now healthy 2024-02-10T09:19:32.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:31.795268+0000 mon.a (mon.0) 285 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:31.795298+0000 mon.a (mon.0) 286 : cluster [INF] osd.1 failed (root=default,host=smithi153) (connection refused reported by osd.0) 2024-02-10T09:19:32.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:31.795460+0000 mon.a (mon.0) 287 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:32.196278+0000 mon.a (mon.0) 288 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:32.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:32 smithi153 bash[35780]: cluster 2024-02-10T09:19:32.196377+0000 mon.a (mon.0) 289 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:31.586296+0000 mon.a (mon.0) 283 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:31.586316+0000 mon.a (mon.0) 284 : cluster [INF] Cluster is now healthy 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:31.795268+0000 mon.a (mon.0) 285 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:31.795298+0000 mon.a (mon.0) 286 : cluster [INF] osd.1 failed (root=default,host=smithi153) (connection refused reported by osd.0) 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:31.795460+0000 mon.a (mon.0) 287 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:32.196278+0000 mon.a (mon.0) 288 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:32 smithi204 bash[35809]: cluster 2024-02-10T09:19:32.196377+0000 mon.a (mon.0) 289 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:19:33.379 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:33 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.1.service: Succeeded. 2024-02-10T09:19:33.379 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:33 smithi153 systemd[1]: Stopped Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:19:33.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:33 smithi153 systemd[1]: Starting Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:19:33.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:33 smithi153 podman[45013]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T09:19:33.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:33 smithi153 systemd[1]: Started Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:19:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: cluster 2024-02-10T09:19:32.595150+0000 mon.a (mon.0) 290 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:19:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: cluster 2024-02-10T09:19:32.604603+0000 mgr.y (mgr.14140) 81 : cluster [DBG] pgmap v75: 1 pgs: 1 stale+active+undersized; 0 B data, 348 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: cluster 2024-02-10T09:19:32.604810+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e21: 3 total, 1 up, 2 in 2024-02-10T09:19:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:32.604951+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: cluster 2024-02-10T09:19:32.595150+0000 mon.a (mon.0) 290 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: cluster 2024-02-10T09:19:32.604603+0000 mgr.y (mgr.14140) 81 : cluster [DBG] pgmap v75: 1 pgs: 1 stale+active+undersized; 0 B data, 348 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: cluster 2024-02-10T09:19:32.604810+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e21: 3 total, 1 up, 2 in 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:32.604951+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:33.550201+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:33.550553+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:33.553361+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:33.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:33.554502+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:33.555598+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[39117]: audit 2024-02-10T09:19:33.556393+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.2", "key": "container_image"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:33.550201+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:33.550553+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:33.553361+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:33.554502+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:33.555598+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:33.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:33 smithi153 bash[35780]: audit 2024-02-10T09:19:33.556393+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.2", "key": "container_image"}]: dispatch 2024-02-10T09:19:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: cluster 2024-02-10T09:19:32.595150+0000 mon.a (mon.0) 290 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:19:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: cluster 2024-02-10T09:19:32.604603+0000 mgr.y (mgr.14140) 81 : cluster [DBG] pgmap v75: 1 pgs: 1 stale+active+undersized; 0 B data, 348 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: cluster 2024-02-10T09:19:32.604810+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e21: 3 total, 1 up, 2 in 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:32.604951+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:33.550201+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:33.550553+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:33.553361+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:33.554502+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:33.555598+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:34.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:33 smithi204 bash[35809]: audit 2024-02-10T09:19:33.556393+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.2", "key": "container_image"}]: dispatch 2024-02-10T09:19:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[35780]: cephadm 2024-02-10T09:19:33.556132+0000 mgr.y (mgr.14140) 82 : cephadm [INF] Deploying daemon osd.2 on smithi153 2024-02-10T09:19:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[35780]: cluster 2024-02-10T09:19:33.610712+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e22: 3 total, 1 up, 2 in 2024-02-10T09:19:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[35780]: audit 2024-02-10T09:19:33.610842+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[39117]: cephadm 2024-02-10T09:19:33.556132+0000 mgr.y (mgr.14140) 82 : cephadm [INF] Deploying daemon osd.2 on smithi153 2024-02-10T09:19:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[39117]: cluster 2024-02-10T09:19:33.610712+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e22: 3 total, 1 up, 2 in 2024-02-10T09:19:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[39117]: audit 2024-02-10T09:19:33.610842+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T09:19:34.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:34 smithi153 bash[45029]: --> ceph-volume lvm activate successful for osd ID: 1 2024-02-10T09:19:35.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:34 smithi204 bash[35809]: cephadm 2024-02-10T09:19:33.556132+0000 mgr.y (mgr.14140) 82 : cephadm [INF] Deploying daemon osd.2 on smithi153 2024-02-10T09:19:35.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:34 smithi204 bash[35809]: cluster 2024-02-10T09:19:33.610712+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e22: 3 total, 1 up, 2 in 2024-02-10T09:19:35.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:34 smithi204 bash[35809]: audit 2024-02-10T09:19:33.610842+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:35 smithi153 bash[35780]: cluster 2024-02-10T09:19:34.604927+0000 mgr.y (mgr.14140) 83 : cluster [DBG] pgmap v77: 1 pgs: 1 stale+active+undersized; 0 B data, 348 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:35 smithi153 bash[39117]: cluster 2024-02-10T09:19:34.604927+0000 mgr.y (mgr.14140) 83 : cluster [DBG] pgmap v77: 1 pgs: 1 stale+active+undersized; 0 B data, 348 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:36.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:35 smithi204 bash[35809]: cluster 2024-02-10T09:19:34.604927+0000 mgr.y (mgr.14140) 83 : cluster [DBG] pgmap v77: 1 pgs: 1 stale+active+undersized; 0 B data, 348 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:37.033 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1,2 on host 'smithi153' 2024-02-10T09:19:37.264 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[45199]: debug 2024-02-10T09:19:37.033+0000 7f95ce6deec0 -1 Falling back to public interface 2024-02-10T09:19:37.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:37 smithi204 bash[35809]: cluster 2024-02-10T09:19:36.605241+0000 mgr.y (mgr.14140) 84 : cluster [DBG] pgmap v78: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:37.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:37 smithi204 bash[35809]: audit 2024-02-10T09:19:37.027573+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:37.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:37 smithi204 bash[35809]: audit 2024-02-10T09:19:37.030812+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:37.857 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[45199]: debug 2024-02-10T09:19:37.589+0000 7f95ce6deec0 -1 osd.1 20 log_to_monitors {default=true} 2024-02-10T09:19:37.857 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[45199]: debug 2024-02-10T09:19:37.595+0000 7f95c26af700 -1 osd.1 20 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:19:37.857 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[35780]: cluster 2024-02-10T09:19:36.605241+0000 mgr.y (mgr.14140) 84 : cluster [DBG] pgmap v78: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:37.857 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[35780]: audit 2024-02-10T09:19:37.027573+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:37.857 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[35780]: audit 2024-02-10T09:19:37.030812+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:37.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[39117]: cluster 2024-02-10T09:19:36.605241+0000 mgr.y (mgr.14140) 84 : cluster [DBG] pgmap v78: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:37.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[39117]: audit 2024-02-10T09:19:37.027573+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:37.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:37 smithi153 bash[39117]: audit 2024-02-10T09:19:37.030812+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077047\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077137\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077168\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:24.077195\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_device_update\": \"2024-02-10T09:19:21.805321\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:37.870 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T09:19:37.872 INFO:tasks.cephadm:Deploying osd.3 on smithi153 with /dev/vg_nvme/lv_1... 2024-02-10T09:19:37.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T09:19:37.902 INFO:journalctl@ceph.osd.2.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T09:19:38.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[45407]: --> ceph-volume lvm activate successful for osd ID: 2 2024-02-10T09:19:38.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[35780]: cluster 2024-02-10T09:19:37.568033+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: Degraded data redundancy: 2/3 objects degraded (66.667%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:19:38.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[35780]: audit 2024-02-10T09:19:37.593107+0000 mon.a (mon.0) 304 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:38.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[35780]: audit 2024-02-10T09:19:37.593440+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[39117]: cluster 2024-02-10T09:19:37.568033+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: Degraded data redundancy: 2/3 objects degraded (66.667%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:19:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[39117]: audit 2024-02-10T09:19:37.593107+0000 mon.a (mon.0) 304 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[39117]: audit 2024-02-10T09:19:37.593440+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:38.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[35780]: audit 2024-02-10T09:19:38.141422+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:38.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[35780]: audit 2024-02-10T09:19:38.144058+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:38.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[35780]: audit 2024-02-10T09:19:38.144611+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[39117]: audit 2024-02-10T09:19:38.141422+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:38.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[39117]: audit 2024-02-10T09:19:38.144058+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:38.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:38 smithi153 bash[39117]: audit 2024-02-10T09:19:38.144611+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:38.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:38 smithi204 bash[35809]: cluster 2024-02-10T09:19:37.568033+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: Degraded data redundancy: 2/3 objects degraded (66.667%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:19:38.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:38 smithi204 bash[35809]: audit 2024-02-10T09:19:37.593107+0000 mon.a (mon.0) 304 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:19:38.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:38 smithi204 bash[35809]: audit 2024-02-10T09:19:37.593440+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:38.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:38 smithi204 bash[35809]: audit 2024-02-10T09:19:38.141422+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:38.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:38 smithi204 bash[35809]: audit 2024-02-10T09:19:38.144058+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:38.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:38 smithi204 bash[35809]: audit 2024-02-10T09:19:38.144611+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:39.834 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[35780]: cluster 2024-02-10T09:19:38.572939+0000 mon.a (mon.0) 309 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[35780]: cluster 2024-02-10T09:19:38.576906+0000 mon.a (mon.0) 310 : cluster [INF] osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780] boot 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[35780]: cluster 2024-02-10T09:19:38.576982+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e23: 3 total, 2 up, 2 in 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[35780]: audit 2024-02-10T09:19:38.577621+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[35780]: audit 2024-02-10T09:19:38.577802+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[35780]: cluster 2024-02-10T09:19:38.605679+0000 mgr.y (mgr.14140) 85 : cluster [DBG] pgmap v80: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[39117]: cluster 2024-02-10T09:19:38.572939+0000 mon.a (mon.0) 309 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[39117]: cluster 2024-02-10T09:19:38.576906+0000 mon.a (mon.0) 310 : cluster [INF] osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780] boot 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[39117]: cluster 2024-02-10T09:19:38.576982+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e23: 3 total, 2 up, 2 in 2024-02-10T09:19:39.835 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[39117]: audit 2024-02-10T09:19:38.577621+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:39.836 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[39117]: audit 2024-02-10T09:19:38.577802+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:39.836 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[39117]: cluster 2024-02-10T09:19:38.605679+0000 mgr.y (mgr.14140) 85 : cluster [DBG] pgmap v80: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:39 smithi204 bash[35809]: cluster 2024-02-10T09:19:38.572939+0000 mon.a (mon.0) 309 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:19:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:39 smithi204 bash[35809]: cluster 2024-02-10T09:19:38.576906+0000 mon.a (mon.0) 310 : cluster [INF] osd.1 [v2:172.21.15.153:6810/1003292780,v1:172.21.15.153:6811/1003292780] boot 2024-02-10T09:19:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:39 smithi204 bash[35809]: cluster 2024-02-10T09:19:38.576982+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e23: 3 total, 2 up, 2 in 2024-02-10T09:19:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:39 smithi204 bash[35809]: audit 2024-02-10T09:19:38.577621+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:19:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:39 smithi204 bash[35809]: audit 2024-02-10T09:19:38.577802+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:39 smithi204 bash[35809]: cluster 2024-02-10T09:19:38.605679+0000 mgr.y (mgr.14140) 85 : cluster [DBG] pgmap v80: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:40.087 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:39 smithi153 bash[45769]: debug 2024-02-10T09:19:39.833+0000 7f15066faec0 -1 Falling back to public interface 2024-02-10T09:19:40.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[45769]: debug 2024-02-10T09:19:40.396+0000 7f15066faec0 -1 osd.2 0 log_to_monitors {default=true} 2024-02-10T09:19:40.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[35780]: cluster 2024-02-10T09:19:39.586156+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e24: 3 total, 2 up, 2 in 2024-02-10T09:19:40.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[35780]: audit 2024-02-10T09:19:39.587835+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:40.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[35780]: audit 2024-02-10T09:19:40.399278+0000 mon.a (mon.0) 316 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:19:40.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[39117]: cluster 2024-02-10T09:19:39.586156+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e24: 3 total, 2 up, 2 in 2024-02-10T09:19:40.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[39117]: audit 2024-02-10T09:19:39.587835+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:40.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:40 smithi153 bash[39117]: audit 2024-02-10T09:19:40.399278+0000 mon.a (mon.0) 316 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:19:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:40 smithi204 bash[35809]: cluster 2024-02-10T09:19:39.586156+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e24: 3 total, 2 up, 2 in 2024-02-10T09:19:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:40 smithi204 bash[35809]: audit 2024-02-10T09:19:39.587835+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:40 smithi204 bash[35809]: audit 2024-02-10T09:19:40.399278+0000 mon.a (mon.0) 316 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:19:41.868 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[45769]: debug 2024-02-10T09:19:41.591+0000 7f14ef4af700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T09:19:41.868 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[45769]: debug 2024-02-10T09:19:41.598+0000 7f14fa6cb700 -1 osd.2 26 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:19:41.868 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: audit 2024-02-10T09:19:40.586670+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T09:19:41.868 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: cluster 2024-02-10T09:19:40.586714+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e25: 3 total, 2 up, 2 in 2024-02-10T09:19:41.868 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: audit 2024-02-10T09:19:40.587118+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:41.868 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: audit 2024-02-10T09:19:40.587399+0000 mon.a (mon.0) 320 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:41.868 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: cluster 2024-02-10T09:19:40.606025+0000 mgr.y (mgr.14140) 86 : cluster [DBG] pgmap v83: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: cluster 2024-02-10T09:19:41.394793+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[35780]: cluster 2024-02-10T09:19:41.394859+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: audit 2024-02-10T09:19:40.586670+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: cluster 2024-02-10T09:19:40.586714+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e25: 3 total, 2 up, 2 in 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: audit 2024-02-10T09:19:40.587118+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: audit 2024-02-10T09:19:40.587399+0000 mon.a (mon.0) 320 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: cluster 2024-02-10T09:19:40.606025+0000 mgr.y (mgr.14140) 86 : cluster [DBG] pgmap v83: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: cluster 2024-02-10T09:19:41.394793+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:19:41.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:41 smithi153 bash[39117]: cluster 2024-02-10T09:19:41.394859+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: audit 2024-02-10T09:19:40.586670+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: cluster 2024-02-10T09:19:40.586714+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e25: 3 total, 2 up, 2 in 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: audit 2024-02-10T09:19:40.587118+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: audit 2024-02-10T09:19:40.587399+0000 mon.a (mon.0) 320 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: cluster 2024-02-10T09:19:40.606025+0000 mgr.y (mgr.14140) 86 : cluster [DBG] pgmap v83: 1 pgs: 1 undersized+degraded+peered; 0 B data, 364 KiB used, 177 GiB / 179 GiB avail; 2/3 objects degraded (66.667%) 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: cluster 2024-02-10T09:19:41.394793+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:19:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:41 smithi204 bash[35809]: cluster 2024-02-10T09:19:41.394859+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:19:42.293 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T09:19:42.293 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T09:19:42.293 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-02-10T09:19:42.293 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-02-10T09:19:42.293 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124571 s, 842 MB/s 2024-02-10T09:19:42.293 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-02-10T09:19:42.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:42 smithi153 bash[35780]: audit 2024-02-10T09:19:41.588662+0000 mon.a (mon.0) 321 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:42.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:42 smithi153 bash[35780]: cluster 2024-02-10T09:19:41.588704+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e26: 3 total, 2 up, 2 in 2024-02-10T09:19:42.909 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:42 smithi153 bash[35780]: audit 2024-02-10T09:19:41.588810+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:42.909 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:42 smithi153 bash[39117]: audit 2024-02-10T09:19:41.588662+0000 mon.a (mon.0) 321 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:42.910 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:42 smithi153 bash[39117]: cluster 2024-02-10T09:19:41.588704+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e26: 3 total, 2 up, 2 in 2024-02-10T09:19:42.910 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:42 smithi153 bash[39117]: audit 2024-02-10T09:19:41.588810+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:43.069 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-02-10T09:19:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:42 smithi204 bash[35809]: audit 2024-02-10T09:19:41.588662+0000 mon.a (mon.0) 321 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:19:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:42 smithi204 bash[35809]: cluster 2024-02-10T09:19:41.588704+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e26: 3 total, 2 up, 2 in 2024-02-10T09:19:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:42 smithi204 bash[35809]: audit 2024-02-10T09:19:41.588810+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[39117]: cluster 2024-02-10T09:19:42.606340+0000 mgr.y (mgr.14140) 87 : cluster [DBG] pgmap v86: 1 pgs: 1 active+undersized+degraded; 0 B data, 420 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[39117]: cluster 2024-02-10T09:19:42.608119+0000 mon.a (mon.0) 324 : cluster [INF] osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904] boot 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[39117]: cluster 2024-02-10T09:19:42.608174+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e27: 3 total, 3 up, 3 in 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[39117]: audit 2024-02-10T09:19:42.608607+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[39117]: cluster 2024-02-10T09:19:43.600770+0000 mon.a (mon.0) 327 : cluster [WRN] Health check update: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[39117]: cluster 2024-02-10T09:19:43.606384+0000 mon.a (mon.0) 328 : cluster [DBG] osdmap e28: 3 total, 3 up, 3 in 2024-02-10T09:19:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[35780]: cluster 2024-02-10T09:19:42.606340+0000 mgr.y (mgr.14140) 87 : cluster [DBG] pgmap v86: 1 pgs: 1 active+undersized+degraded; 0 B data, 420 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[35780]: cluster 2024-02-10T09:19:42.608119+0000 mon.a (mon.0) 324 : cluster [INF] osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904] boot 2024-02-10T09:19:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[35780]: cluster 2024-02-10T09:19:42.608174+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e27: 3 total, 3 up, 3 in 2024-02-10T09:19:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[35780]: audit 2024-02-10T09:19:42.608607+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[35780]: cluster 2024-02-10T09:19:43.600770+0000 mon.a (mon.0) 327 : cluster [WRN] Health check update: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:19:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:43 smithi153 bash[35780]: cluster 2024-02-10T09:19:43.606384+0000 mon.a (mon.0) 328 : cluster [DBG] osdmap e28: 3 total, 3 up, 3 in 2024-02-10T09:19:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:43 smithi204 bash[35809]: cluster 2024-02-10T09:19:42.606340+0000 mgr.y (mgr.14140) 87 : cluster [DBG] pgmap v86: 1 pgs: 1 active+undersized+degraded; 0 B data, 420 KiB used, 177 GiB / 179 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:43 smithi204 bash[35809]: cluster 2024-02-10T09:19:42.608119+0000 mon.a (mon.0) 324 : cluster [INF] osd.2 [v2:172.21.15.153:6818/2495526904,v1:172.21.15.153:6819/2495526904] boot 2024-02-10T09:19:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:43 smithi204 bash[35809]: cluster 2024-02-10T09:19:42.608174+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e27: 3 total, 3 up, 3 in 2024-02-10T09:19:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:43 smithi204 bash[35809]: audit 2024-02-10T09:19:42.608607+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:19:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:43 smithi204 bash[35809]: cluster 2024-02-10T09:19:43.600770+0000 mon.a (mon.0) 327 : cluster [WRN] Health check update: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:19:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:43 smithi204 bash[35809]: cluster 2024-02-10T09:19:43.606384+0000 mon.a (mon.0) 328 : cluster [DBG] osdmap e28: 3 total, 3 up, 3 in 2024-02-10T09:19:44.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[35780]: audit 2024-02-10T09:19:43.720019+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:44.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[39117]: audit 2024-02-10T09:19:43.720019+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:44.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[39117]: audit 2024-02-10T09:19:43.903362+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[35780]: audit 2024-02-10T09:19:43.903362+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[35780]: audit 2024-02-10T09:19:43.906109+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[35780]: audit 2024-02-10T09:19:43.907397+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:44.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[39117]: audit 2024-02-10T09:19:43.906109+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:44.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:44 smithi153 bash[39117]: audit 2024-02-10T09:19:43.907397+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:45.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:44 smithi204 bash[35809]: audit 2024-02-10T09:19:43.720019+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:45.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:44 smithi204 bash[35809]: audit 2024-02-10T09:19:43.903362+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:45.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:44 smithi204 bash[35809]: audit 2024-02-10T09:19:43.906109+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140509\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140585\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140617\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140645\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:38.140673\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:38.140752\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:45.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:44 smithi204 bash[35809]: audit 2024-02-10T09:19:43.907397+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:45.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:45 smithi153 bash[39117]: cluster 2024-02-10T09:19:43.906628+0000 mgr.y (mgr.14140) 88 : cluster [DBG] pgmap v88: 1 pgs: 1 active+undersized+degraded; 0 B data, 652 KiB used, 265 GiB / 268 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:45.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:45 smithi153 bash[39117]: cluster 2024-02-10T09:19:44.904577+0000 mon.a (mon.0) 333 : cluster [WRN] Health check failed: 1 stray daemons(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-02-10T09:19:45.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:45 smithi153 bash[35780]: cluster 2024-02-10T09:19:43.906628+0000 mgr.y (mgr.14140) 88 : cluster [DBG] pgmap v88: 1 pgs: 1 active+undersized+degraded; 0 B data, 652 KiB used, 265 GiB / 268 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:45.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:45 smithi153 bash[35780]: cluster 2024-02-10T09:19:44.904577+0000 mon.a (mon.0) 333 : cluster [WRN] Health check failed: 1 stray daemons(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-02-10T09:19:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:45 smithi204 bash[35809]: cluster 2024-02-10T09:19:43.906628+0000 mgr.y (mgr.14140) 88 : cluster [DBG] pgmap v88: 1 pgs: 1 active+undersized+degraded; 0 B data, 652 KiB used, 265 GiB / 268 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:45 smithi204 bash[35809]: cluster 2024-02-10T09:19:44.904577+0000 mon.a (mon.0) 333 : cluster [WRN] Health check failed: 1 stray daemons(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-02-10T09:19:46.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[35780]: audit 2024-02-10T09:19:45.728006+0000 mgr.y (mgr.14140) 89 : audit [DBG] from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[35780]: cephadm 2024-02-10T09:19:45.728587+0000 mgr.y (mgr.14140) 90 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[35780]: audit 2024-02-10T09:19:45.728872+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[35780]: audit 2024-02-10T09:19:45.729450+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[35780]: audit 2024-02-10T09:19:45.730193+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[39117]: audit 2024-02-10T09:19:45.728006+0000 mgr.y (mgr.14140) 89 : audit [DBG] from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[39117]: cephadm 2024-02-10T09:19:45.728587+0000 mgr.y (mgr.14140) 90 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[39117]: audit 2024-02-10T09:19:45.728872+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[39117]: audit 2024-02-10T09:19:45.729450+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:46 smithi153 bash[39117]: audit 2024-02-10T09:19:45.730193+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:47.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:46 smithi204 bash[35809]: audit 2024-02-10T09:19:45.728006+0000 mgr.y (mgr.14140) 89 : audit [DBG] from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:19:47.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:46 smithi204 bash[35809]: cephadm 2024-02-10T09:19:45.728587+0000 mgr.y (mgr.14140) 90 : cephadm [INF] Applying osd on host smithi153... 2024-02-10T09:19:47.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:46 smithi204 bash[35809]: audit 2024-02-10T09:19:45.728872+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:19:47.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:46 smithi204 bash[35809]: audit 2024-02-10T09:19:45.729450+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:47.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:46 smithi204 bash[35809]: audit 2024-02-10T09:19:45.730193+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:47.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:47 smithi153 bash[35780]: cluster 2024-02-10T09:19:45.906954+0000 mgr.y (mgr.14140) 91 : cluster [DBG] pgmap v89: 1 pgs: 1 active+undersized+degraded; 0 B data, 684 KiB used, 265 GiB / 268 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:47.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:47 smithi153 bash[39117]: cluster 2024-02-10T09:19:45.906954+0000 mgr.y (mgr.14140) 91 : cluster [DBG] pgmap v89: 1 pgs: 1 active+undersized+degraded; 0 B data, 684 KiB used, 265 GiB / 268 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:48.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:47 smithi204 bash[35809]: cluster 2024-02-10T09:19:45.906954+0000 mgr.y (mgr.14140) 91 : cluster [DBG] pgmap v89: 1 pgs: 1 active+undersized+degraded; 0 B data, 684 KiB used, 265 GiB / 268 GiB avail; 1/3 objects degraded (33.333%) 2024-02-10T09:19:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: audit 2024-02-10T09:19:47.765621+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:47.764886\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:48.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: audit 2024-02-10T09:19:47.765621+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:47.764886\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: audit 2024-02-10T09:19:47.768594+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:47.764886\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: cluster 2024-02-10T09:19:47.769209+0000 mgr.y (mgr.14140) 92 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 708 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: audit 2024-02-10T09:19:48.026968+0000 mon.a (mon.0) 339 : audit [INF] from='client.? 172.21.15.153:0/1405187811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "679b994c-2b1f-4f4d-9b26-64a3d365a094"}]: dispatch 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: audit 2024-02-10T09:19:48.032278+0000 mon.a (mon.0) 340 : audit [INF] from='client.? 172.21.15.153:0/1405187811' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "679b994c-2b1f-4f4d-9b26-64a3d365a094"}]': finished 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: cluster 2024-02-10T09:19:48.032321+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e29: 4 total, 3 up, 3 in 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[35780]: audit 2024-02-10T09:19:48.032417+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: audit 2024-02-10T09:19:47.768594+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:47.764886\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: cluster 2024-02-10T09:19:47.769209+0000 mgr.y (mgr.14140) 92 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 708 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: audit 2024-02-10T09:19:48.026968+0000 mon.a (mon.0) 339 : audit [INF] from='client.? 172.21.15.153:0/1405187811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "679b994c-2b1f-4f4d-9b26-64a3d365a094"}]: dispatch 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: audit 2024-02-10T09:19:48.032278+0000 mon.a (mon.0) 340 : audit [INF] from='client.? 172.21.15.153:0/1405187811' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "679b994c-2b1f-4f4d-9b26-64a3d365a094"}]': finished 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: cluster 2024-02-10T09:19:48.032321+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e29: 4 total, 3 up, 3 in 2024-02-10T09:19:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:48 smithi153 bash[39117]: audit 2024-02-10T09:19:48.032417+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:19:49.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: audit 2024-02-10T09:19:47.765621+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:47.764886\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: audit 2024-02-10T09:19:47.768594+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}}, \"last_daemon_update\": \"2024-02-10T09:19:47.764886\", \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: cluster 2024-02-10T09:19:47.769209+0000 mgr.y (mgr.14140) 92 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 708 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: audit 2024-02-10T09:19:48.026968+0000 mon.a (mon.0) 339 : audit [INF] from='client.? 172.21.15.153:0/1405187811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "679b994c-2b1f-4f4d-9b26-64a3d365a094"}]: dispatch 2024-02-10T09:19:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: audit 2024-02-10T09:19:48.032278+0000 mon.a (mon.0) 340 : audit [INF] from='client.? 172.21.15.153:0/1405187811' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "679b994c-2b1f-4f4d-9b26-64a3d365a094"}]': finished 2024-02-10T09:19:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: cluster 2024-02-10T09:19:48.032321+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e29: 4 total, 3 up, 3 in 2024-02-10T09:19:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:48 smithi204 bash[35809]: audit 2024-02-10T09:19:48.032417+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[35780]: cluster 2024-02-10T09:19:48.766758+0000 mon.a (mon.0) 343 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemons(s) not managed by cephadm) 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[35780]: cluster 2024-02-10T09:19:48.766780+0000 mon.a (mon.0) 344 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded) 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[35780]: cluster 2024-02-10T09:19:48.766806+0000 mon.a (mon.0) 345 : cluster [INF] Cluster is now healthy 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[35780]: audit 2024-02-10T09:19:48.782807+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.153:0/4224359635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[39117]: cluster 2024-02-10T09:19:48.766758+0000 mon.a (mon.0) 343 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemons(s) not managed by cephadm) 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[39117]: cluster 2024-02-10T09:19:48.766780+0000 mon.a (mon.0) 344 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded) 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[39117]: cluster 2024-02-10T09:19:48.766806+0000 mon.a (mon.0) 345 : cluster [INF] Cluster is now healthy 2024-02-10T09:19:49.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:49 smithi153 bash[39117]: audit 2024-02-10T09:19:48.782807+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.153:0/4224359635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:49 smithi204 bash[35809]: cluster 2024-02-10T09:19:48.766758+0000 mon.a (mon.0) 343 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemons(s) not managed by cephadm) 2024-02-10T09:19:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:49 smithi204 bash[35809]: cluster 2024-02-10T09:19:48.766780+0000 mon.a (mon.0) 344 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded) 2024-02-10T09:19:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:49 smithi204 bash[35809]: cluster 2024-02-10T09:19:48.766806+0000 mon.a (mon.0) 345 : cluster [INF] Cluster is now healthy 2024-02-10T09:19:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:49 smithi204 bash[35809]: audit 2024-02-10T09:19:48.782807+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.153:0/4224359635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:19:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:50 smithi153 bash[35780]: cluster 2024-02-10T09:19:49.769499+0000 mgr.y (mgr.14140) 93 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 716 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:50.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:50 smithi153 bash[39117]: cluster 2024-02-10T09:19:49.769499+0000 mgr.y (mgr.14140) 93 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 716 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:51.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:50 smithi204 bash[35809]: cluster 2024-02-10T09:19:49.769499+0000 mgr.y (mgr.14140) 93 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 716 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:51.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:51 smithi153 bash[35780]: audit 2024-02-10T09:19:51.231845+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:51.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:51 smithi153 bash[39117]: audit 2024-02-10T09:19:51.231845+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:51 smithi204 bash[35809]: audit 2024-02-10T09:19:51.231845+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:52.725 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:52 smithi153 bash[39117]: cluster 2024-02-10T09:19:51.769793+0000 mgr.y (mgr.14140) 94 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 724 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:52.726 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:52 smithi153 bash[35780]: cluster 2024-02-10T09:19:51.769793+0000 mgr.y (mgr.14140) 94 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 724 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:52 smithi204 bash[35809]: cluster 2024-02-10T09:19:51.769793+0000 mgr.y (mgr.14140) 94 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 724 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[35780]: cluster 2024-02-10T09:19:53.770061+0000 mgr.y (mgr.14140) 95 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[35780]: audit 2024-02-10T09:19:53.783840+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[35780]: audit 2024-02-10T09:19:53.784360+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[35780]: cephadm 2024-02-10T09:19:53.784751+0000 mgr.y (mgr.14140) 96 : cephadm [INF] Deploying daemon osd.3 on smithi153 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[35780]: audit 2024-02-10T09:19:53.784934+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.3", "key": "container_image"}]: dispatch 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[39117]: cluster 2024-02-10T09:19:53.770061+0000 mgr.y (mgr.14140) 95 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[39117]: audit 2024-02-10T09:19:53.783840+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:19:54.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[39117]: audit 2024-02-10T09:19:53.784360+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:54.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[39117]: cephadm 2024-02-10T09:19:53.784751+0000 mgr.y (mgr.14140) 96 : cephadm [INF] Deploying daemon osd.3 on smithi153 2024-02-10T09:19:54.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:53 smithi153 bash[39117]: audit 2024-02-10T09:19:53.784934+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.3", "key": "container_image"}]: dispatch 2024-02-10T09:19:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:53 smithi204 bash[35809]: cluster 2024-02-10T09:19:53.770061+0000 mgr.y (mgr.14140) 95 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:53 smithi204 bash[35809]: audit 2024-02-10T09:19:53.783840+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:19:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:53 smithi204 bash[35809]: audit 2024-02-10T09:19:53.784360+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:19:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:53 smithi204 bash[35809]: cephadm 2024-02-10T09:19:53.784751+0000 mgr.y (mgr.14140) 96 : cephadm [INF] Deploying daemon osd.3 on smithi153 2024-02-10T09:19:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:53 smithi204 bash[35809]: audit 2024-02-10T09:19:53.784934+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.3", "key": "container_image"}]: dispatch 2024-02-10T09:19:55.349 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 3 on host 'smithi153' 2024-02-10T09:19:55.598 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:55 smithi153 bash[39117]: audit 2024-02-10T09:19:55.342642+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:55.598 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:55 smithi153 bash[39117]: audit 2024-02-10T09:19:55.343058+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:55.598 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:55 smithi153 bash[39117]: audit 2024-02-10T09:19:55.346063+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:55.599 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:55 smithi153 bash[35780]: audit 2024-02-10T09:19:55.342642+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:55.599 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:55 smithi153 bash[35780]: audit 2024-02-10T09:19:55.343058+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:55.599 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:55 smithi153 bash[35780]: audit 2024-02-10T09:19:55.346063+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:55.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:55 smithi204 bash[35809]: audit 2024-02-10T09:19:55.342642+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:55.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:55 smithi204 bash[35809]: audit 2024-02-10T09:19:55.343058+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:19:55.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:55 smithi204 bash[35809]: audit 2024-02-10T09:19:55.346063+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764605\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764676\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764706\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764758\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764787\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:47.764816\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_device_update\": \"2024-02-10T09:19:43.902777\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:56.402 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:56 smithi153 bash[35780]: cluster 2024-02-10T09:19:55.770331+0000 mgr.y (mgr.14140) 97 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:56.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:56 smithi153 bash[39117]: cluster 2024-02-10T09:19:55.770331+0000 mgr.y (mgr.14140) 97 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:56.788 DEBUG:teuthology.orchestra.run.smithi153:osd.3> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T09:19:56.791 INFO:tasks.cephadm:Deploying osd.4 on smithi204 with /dev/vg_nvme/lv_4... 2024-02-10T09:19:56.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T09:19:56.820 INFO:journalctl@ceph.osd.3.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:19:56.824 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:56 smithi204 bash[35809]: cluster 2024-02-10T09:19:55.770331+0000 mgr.y (mgr.14140) 97 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T09:19:57.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[47177]: --> ceph-volume lvm activate successful for osd ID: 3 2024-02-10T09:19:58.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:57 smithi204 bash[35809]: cluster 2024-02-10T09:19:57.770664+0000 mgr.y (mgr.14140) 98 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:19:58.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[39117]: cluster 2024-02-10T09:19:57.770664+0000 mgr.y (mgr.14140) 98 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:19:58.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:57 smithi153 bash[35780]: cluster 2024-02-10T09:19:57.770664+0000 mgr.y (mgr.14140) 98 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:19:58.639 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T09:19:58.639 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T09:19:58.639 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-02-10T09:19:58.640 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-02-10T09:19:58.640 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109997 s, 953 MB/s 2024-02-10T09:19:58.640 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-02-10T09:19:59.021 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[47529]: debug 2024-02-10T09:19:58.802+0000 7f9c0cfbbec0 -1 Falling back to public interface 2024-02-10T09:19:59.022 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[35780]: audit 2024-02-10T09:19:58.375266+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:59.022 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[35780]: audit 2024-02-10T09:19:58.378043+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:59.022 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[35780]: audit 2024-02-10T09:19:58.378547+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:59.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[39117]: audit 2024-02-10T09:19:58.375266+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:59.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[39117]: audit 2024-02-10T09:19:58.378043+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:59.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:58 smithi153 bash[39117]: audit 2024-02-10T09:19:58.378547+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:58 smithi204 bash[35809]: audit 2024-02-10T09:19:58.375266+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:19:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:58 smithi204 bash[35809]: audit 2024-02-10T09:19:58.378043+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"locked\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:19:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:58 smithi204 bash[35809]: audit 2024-02-10T09:19:58.378547+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:19:59.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-02-10T09:19:59.688 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[47529]: debug 2024-02-10T09:19:59.364+0000 7f9c0cfbbec0 -1 osd.3 0 log_to_monitors {default=true} 2024-02-10T09:19:59.987 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[35780]: audit 2024-02-10T09:19:59.367601+0000 mon.c (mon.1) 10 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:19:59.987 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[35780]: audit 2024-02-10T09:19:59.367883+0000 mon.a (mon.0) 356 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:19:59.987 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[35780]: cluster 2024-02-10T09:19:59.771062+0000 mgr.y (mgr.14140) 99 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:19:59.987 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[39117]: audit 2024-02-10T09:19:59.367601+0000 mon.c (mon.1) 10 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:19:59.987 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[39117]: audit 2024-02-10T09:19:59.367883+0000 mon.a (mon.0) 356 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:19:59.988 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:19:59 smithi153 bash[39117]: cluster 2024-02-10T09:19:59.771062+0000 mgr.y (mgr.14140) 99 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:20:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:59 smithi204 bash[35809]: audit 2024-02-10T09:19:59.367601+0000 mon.c (mon.1) 10 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:20:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:59 smithi204 bash[35809]: audit 2024-02-10T09:19:59.367883+0000 mon.a (mon.0) 356 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:20:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:19:59 smithi204 bash[35809]: cluster 2024-02-10T09:19:59.771062+0000 mgr.y (mgr.14140) 99 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:20:01.060 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[47529]: debug 2024-02-10T09:20:00.880+0000 7f9bf5d70700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T09:20:01.060 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[47529]: debug 2024-02-10T09:20:00.884+0000 7f9c00f8c700 -1 osd.3 31 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:20:01.060 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: audit 2024-02-10T09:19:59.875090+0000 mon.a (mon.0) 357 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T09:20:01.060 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: cluster 2024-02-10T09:19:59.875132+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e30: 4 total, 3 up, 3 in 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: audit 2024-02-10T09:19:59.875247+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: audit 2024-02-10T09:19:59.876598+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: audit 2024-02-10T09:19:59.876824+0000 mon.a (mon.0) 360 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: cluster 2024-02-10T09:20:00.327815+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[35780]: cluster 2024-02-10T09:20:00.327854+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: audit 2024-02-10T09:19:59.875090+0000 mon.a (mon.0) 357 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: cluster 2024-02-10T09:19:59.875132+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e30: 4 total, 3 up, 3 in 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: audit 2024-02-10T09:19:59.875247+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: audit 2024-02-10T09:19:59.876598+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: audit 2024-02-10T09:19:59.876824+0000 mon.a (mon.0) 360 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: cluster 2024-02-10T09:20:00.327815+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:00 smithi153 bash[39117]: cluster 2024-02-10T09:20:00.327854+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: audit 2024-02-10T09:19:59.875090+0000 mon.a (mon.0) 357 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: cluster 2024-02-10T09:19:59.875132+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e30: 4 total, 3 up, 3 in 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: audit 2024-02-10T09:19:59.875247+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: audit 2024-02-10T09:19:59.876598+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: audit 2024-02-10T09:19:59.876824+0000 mon.a (mon.0) 360 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: cluster 2024-02-10T09:20:00.327815+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:00 smithi204 bash[35809]: cluster 2024-02-10T09:20:00.327854+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:00.876190+0000 mon.a (mon.0) 361 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: cluster 2024-02-10T09:20:00.876238+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e31: 4 total, 3 up, 3 in 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:00.876722+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.139654+0000 mgr.y (mgr.14140) 100 : audit [DBG] from='client.34199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: cephadm 2024-02-10T09:20:01.140380+0000 mgr.y (mgr.14140) 101 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.140667+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.141167+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.141870+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.391595+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:00.876190+0000 mon.a (mon.0) 361 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: cluster 2024-02-10T09:20:00.876238+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e31: 4 total, 3 up, 3 in 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:00.876722+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.139654+0000 mgr.y (mgr.14140) 100 : audit [DBG] from='client.34199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: cephadm 2024-02-10T09:20:01.140380+0000 mgr.y (mgr.14140) 101 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.140667+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.141167+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.141870+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.391595+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.572679+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: audit 2024-02-10T09:20:01.576946+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[39117]: cluster 2024-02-10T09:20:01.771352+0000 mgr.y (mgr.14140) 102 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.572679+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:20:02.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: audit 2024-02-10T09:20:01.576946+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:20:02.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:01 smithi153 bash[35780]: cluster 2024-02-10T09:20:01.771352+0000 mgr.y (mgr.14140) 102 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:20:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:00.876190+0000 mon.a (mon.0) 361 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-02-10T09:20:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: cluster 2024-02-10T09:20:00.876238+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e31: 4 total, 3 up, 3 in 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:00.876722+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.139654+0000 mgr.y (mgr.14140) 100 : audit [DBG] from='client.34199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: cephadm 2024-02-10T09:20:01.140380+0000 mgr.y (mgr.14140) 101 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.140667+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.141167+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.141870+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.391595+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.572679+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: audit 2024-02-10T09:20:01.576946+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}}, \"last_daemon_update\": \"2024-02-10T09:19:58.374555\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:20:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:01 smithi204 bash[35809]: cluster 2024-02-10T09:20:01.771352+0000 mgr.y (mgr.14140) 102 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 740 KiB used, 265 GiB / 268 GiB avail 2024-02-10T09:20:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:02 smithi153 bash[39117]: cluster 2024-02-10T09:20:01.882124+0000 mon.a (mon.0) 370 : cluster [INF] osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920] boot 2024-02-10T09:20:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:02 smithi153 bash[39117]: cluster 2024-02-10T09:20:01.882172+0000 mon.a (mon.0) 371 : cluster [DBG] osdmap e32: 4 total, 4 up, 4 in 2024-02-10T09:20:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:02 smithi153 bash[39117]: audit 2024-02-10T09:20:01.882659+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:03.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:02 smithi153 bash[35780]: cluster 2024-02-10T09:20:01.882124+0000 mon.a (mon.0) 370 : cluster [INF] osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920] boot 2024-02-10T09:20:03.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:02 smithi153 bash[35780]: cluster 2024-02-10T09:20:01.882172+0000 mon.a (mon.0) 371 : cluster [DBG] osdmap e32: 4 total, 4 up, 4 in 2024-02-10T09:20:03.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:02 smithi153 bash[35780]: audit 2024-02-10T09:20:01.882659+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:02 smithi204 bash[35809]: cluster 2024-02-10T09:20:01.882124+0000 mon.a (mon.0) 370 : cluster [INF] osd.3 [v2:172.21.15.153:6826/918480920,v1:172.21.15.153:6827/918480920] boot 2024-02-10T09:20:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:02 smithi204 bash[35809]: cluster 2024-02-10T09:20:01.882172+0000 mon.a (mon.0) 371 : cluster [DBG] osdmap e32: 4 total, 4 up, 4 in 2024-02-10T09:20:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:02 smithi204 bash[35809]: audit 2024-02-10T09:20:01.882659+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: cluster 2024-02-10T09:20:02.892369+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e33: 4 total, 4 up, 4 in 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: audit 2024-02-10T09:20:03.295814+0000 mon.b (mon.2) 4 : audit [INF] from='client.? 172.21.15.204:0/622658850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]: dispatch 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: audit 2024-02-10T09:20:03.296163+0000 mon.a (mon.0) 374 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]: dispatch 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: audit 2024-02-10T09:20:03.300888+0000 mon.a (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]': finished 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: cluster 2024-02-10T09:20:03.300935+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e34: 5 total, 4 up, 4 in 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: audit 2024-02-10T09:20:03.301087+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[35780]: cluster 2024-02-10T09:20:03.771609+0000 mgr.y (mgr.14140) 103 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 1.0 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: cluster 2024-02-10T09:20:02.892369+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e33: 4 total, 4 up, 4 in 2024-02-10T09:20:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: audit 2024-02-10T09:20:03.295814+0000 mon.b (mon.2) 4 : audit [INF] from='client.? 172.21.15.204:0/622658850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]: dispatch 2024-02-10T09:20:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: audit 2024-02-10T09:20:03.296163+0000 mon.a (mon.0) 374 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]: dispatch 2024-02-10T09:20:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: audit 2024-02-10T09:20:03.300888+0000 mon.a (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]': finished 2024-02-10T09:20:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: cluster 2024-02-10T09:20:03.300935+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e34: 5 total, 4 up, 4 in 2024-02-10T09:20:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: audit 2024-02-10T09:20:03.301087+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:03 smithi153 bash[39117]: cluster 2024-02-10T09:20:03.771609+0000 mgr.y (mgr.14140) 103 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 1.0 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: cluster 2024-02-10T09:20:02.892369+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e33: 4 total, 4 up, 4 in 2024-02-10T09:20:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: audit 2024-02-10T09:20:03.295814+0000 mon.b (mon.2) 4 : audit [INF] from='client.? 172.21.15.204:0/622658850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]: dispatch 2024-02-10T09:20:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: audit 2024-02-10T09:20:03.296163+0000 mon.a (mon.0) 374 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]: dispatch 2024-02-10T09:20:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: audit 2024-02-10T09:20:03.300888+0000 mon.a (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e6b26a2-2084-4ce3-a544-5f4e186ff1a0"}]': finished 2024-02-10T09:20:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: cluster 2024-02-10T09:20:03.300935+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e34: 5 total, 4 up, 4 in 2024-02-10T09:20:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: audit 2024-02-10T09:20:03.301087+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:03 smithi204 bash[35809]: cluster 2024-02-10T09:20:03.771609+0000 mgr.y (mgr.14140) 103 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 1.0 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:04 smithi153 bash[35780]: audit 2024-02-10T09:20:04.072510+0000 mon.b (mon.2) 5 : audit [DBG] from='client.? 172.21.15.204:0/877795508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:04 smithi153 bash[39117]: audit 2024-02-10T09:20:04.072510+0000 mon.b (mon.2) 5 : audit [DBG] from='client.? 172.21.15.204:0/877795508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:04 smithi204 bash[35809]: audit 2024-02-10T09:20:04.072510+0000 mon.b (mon.2) 5 : audit [DBG] from='client.? 172.21.15.204:0/877795508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:05 smithi153 bash[35780]: cluster 2024-02-10T09:20:05.771918+0000 mgr.y (mgr.14140) 104 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.0 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:05 smithi153 bash[39117]: cluster 2024-02-10T09:20:05.771918+0000 mgr.y (mgr.14140) 104 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.0 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:05 smithi204 bash[35809]: cluster 2024-02-10T09:20:05.771918+0000 mgr.y (mgr.14140) 104 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.0 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:07.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:07 smithi204 bash[35809]: audit 2024-02-10T09:20:06.629025+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:07.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:07 smithi153 bash[35780]: audit 2024-02-10T09:20:06.629025+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:07 smithi153 bash[39117]: audit 2024-02-10T09:20:06.629025+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:08.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:08 smithi204 bash[35809]: cluster 2024-02-10T09:20:07.772287+0000 mgr.y (mgr.14140) 105 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:08.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:08 smithi153 bash[35780]: cluster 2024-02-10T09:20:07.772287+0000 mgr.y (mgr.14140) 105 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:08.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:08 smithi153 bash[39117]: cluster 2024-02-10T09:20:07.772287+0000 mgr.y (mgr.14140) 105 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:09.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:09 smithi204 bash[35809]: audit 2024-02-10T09:20:08.905645+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:20:09.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:09 smithi204 bash[35809]: audit 2024-02-10T09:20:08.906394+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:09.861 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:09 smithi204 bash[35809]: audit 2024-02-10T09:20:08.907207+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.4", "key": "container_image"}]: dispatch 2024-02-10T09:20:09.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:09 smithi153 bash[35780]: audit 2024-02-10T09:20:08.905645+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:20:09.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:09 smithi153 bash[35780]: audit 2024-02-10T09:20:08.906394+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:09.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:09 smithi153 bash[35780]: audit 2024-02-10T09:20:08.907207+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.4", "key": "container_image"}]: dispatch 2024-02-10T09:20:09.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:09 smithi153 bash[39117]: audit 2024-02-10T09:20:08.905645+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:20:09.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:09 smithi153 bash[39117]: audit 2024-02-10T09:20:08.906394+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:09.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:09 smithi153 bash[39117]: audit 2024-02-10T09:20:08.907207+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.4", "key": "container_image"}]: dispatch 2024-02-10T09:20:10.482 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 4 on host 'smithi204' 2024-02-10T09:20:10.713 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:10 smithi204 bash[35809]: cephadm 2024-02-10T09:20:08.906952+0000 mgr.y (mgr.14140) 106 : cephadm [INF] Deploying daemon osd.4 on smithi204 2024-02-10T09:20:10.713 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:10 smithi204 bash[35809]: cluster 2024-02-10T09:20:09.772647+0000 mgr.y (mgr.14140) 107 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:10.713 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:10 smithi204 bash[35809]: audit 2024-02-10T09:20:10.475608+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:10.713 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:10 smithi204 bash[35809]: audit 2024-02-10T09:20:10.476053+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:10.713 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:10 smithi204 bash[35809]: audit 2024-02-10T09:20:10.478935+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:10.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[35780]: cephadm 2024-02-10T09:20:08.906952+0000 mgr.y (mgr.14140) 106 : cephadm [INF] Deploying daemon osd.4 on smithi204 2024-02-10T09:20:10.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[35780]: cluster 2024-02-10T09:20:09.772647+0000 mgr.y (mgr.14140) 107 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:10.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[35780]: audit 2024-02-10T09:20:10.475608+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:10.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[39117]: cephadm 2024-02-10T09:20:08.906952+0000 mgr.y (mgr.14140) 106 : cephadm [INF] Deploying daemon osd.4 on smithi204 2024-02-10T09:20:10.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[39117]: cluster 2024-02-10T09:20:09.772647+0000 mgr.y (mgr.14140) 107 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:10.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[39117]: audit 2024-02-10T09:20:10.475608+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:10.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[35780]: audit 2024-02-10T09:20:10.476053+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:10.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[35780]: audit 2024-02-10T09:20:10.478935+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:10.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[39117]: audit 2024-02-10T09:20:10.476053+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:10.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:10 smithi153 bash[39117]: audit 2024-02-10T09:20:10.478935+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157262\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:18:37.157340\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_device_update\": \"2024-02-10T09:17:54.870035\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:12.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:12 smithi204 bash[35809]: cluster 2024-02-10T09:20:11.773099+0000 mgr.y (mgr.14140) 108 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:12.870 DEBUG:teuthology.orchestra.run.smithi204:osd.4> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T09:20:12.872 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-02-10T09:20:12.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T09:20:12.902 INFO:journalctl@ceph.osd.4.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:20:12.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:12 smithi153 bash[35780]: cluster 2024-02-10T09:20:11.773099+0000 mgr.y (mgr.14140) 108 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:12.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:12 smithi153 bash[39117]: cluster 2024-02-10T09:20:11.773099+0000 mgr.y (mgr.14140) 108 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:13.664 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:20:13 smithi204 bash[38176]: debug 2024-02-10T09:20:13.409+0000 7f779dda1ec0 -1 osd.4 0 log_to_monitors {default=true} 2024-02-10T09:20:13.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:13 smithi204 bash[35809]: audit 2024-02-10T09:20:13.413424+0000 mon.b (mon.2) 6 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:20:13.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:13 smithi204 bash[35809]: audit 2024-02-10T09:20:13.413792+0000 mon.a (mon.0) 385 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:20:13.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:13 smithi204 bash[35809]: audit 2024-02-10T09:20:13.527849+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:13.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:13 smithi204 bash[35809]: audit 2024-02-10T09:20:13.531369+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:13.664 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:13 smithi204 bash[35809]: audit 2024-02-10T09:20:13.532039+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:13.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[35780]: audit 2024-02-10T09:20:13.413424+0000 mon.b (mon.2) 6 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:20:13.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[35780]: audit 2024-02-10T09:20:13.413792+0000 mon.a (mon.0) 385 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:20:13.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[39117]: audit 2024-02-10T09:20:13.413424+0000 mon.b (mon.2) 6 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:20:13.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[39117]: audit 2024-02-10T09:20:13.413792+0000 mon.a (mon.0) 385 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:20:13.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[35780]: audit 2024-02-10T09:20:13.527849+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:13.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[35780]: audit 2024-02-10T09:20:13.531369+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:13.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[35780]: audit 2024-02-10T09:20:13.532039+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:13.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[39117]: audit 2024-02-10T09:20:13.527849+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:13.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[39117]: audit 2024-02-10T09:20:13.531369+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"Insufficient space (<5GB) on vgs\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:13.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:13 smithi153 bash[39117]: audit 2024-02-10T09:20:13.532039+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:14.912 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[38176]: debug 2024-02-10T09:20:14.600+0000 7f7782b4e700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T09:20:14.913 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[38176]: debug 2024-02-10T09:20:14.607+0000 7f778dd6a700 -1 osd.4 36 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:20:14.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[35809]: audit 2024-02-10T09:20:13.596698+0000 mon.a (mon.0) 389 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T09:20:14.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[35809]: cluster 2024-02-10T09:20:13.596768+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e35: 5 total, 4 up, 4 in 2024-02-10T09:20:14.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[35809]: audit 2024-02-10T09:20:13.596895+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:14.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[35809]: audit 2024-02-10T09:20:13.597202+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:14.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[35809]: audit 2024-02-10T09:20:13.597425+0000 mon.a (mon.0) 392 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:14.913 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:14 smithi204 bash[35809]: cluster 2024-02-10T09:20:13.773407+0000 mgr.y (mgr.14140) 109 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:14.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[35780]: audit 2024-02-10T09:20:13.596698+0000 mon.a (mon.0) 389 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[35780]: cluster 2024-02-10T09:20:13.596768+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e35: 5 total, 4 up, 4 in 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[35780]: audit 2024-02-10T09:20:13.596895+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[35780]: audit 2024-02-10T09:20:13.597202+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[35780]: audit 2024-02-10T09:20:13.597425+0000 mon.a (mon.0) 392 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[35780]: cluster 2024-02-10T09:20:13.773407+0000 mgr.y (mgr.14140) 109 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[39117]: audit 2024-02-10T09:20:13.596698+0000 mon.a (mon.0) 389 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[39117]: cluster 2024-02-10T09:20:13.596768+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e35: 5 total, 4 up, 4 in 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[39117]: audit 2024-02-10T09:20:13.596895+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[39117]: audit 2024-02-10T09:20:13.597202+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[39117]: audit 2024-02-10T09:20:13.597425+0000 mon.a (mon.0) 392 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:14.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:14 smithi153 bash[39117]: cluster 2024-02-10T09:20:13.773407+0000 mgr.y (mgr.14140) 109 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 MiB used, 354 GiB / 358 GiB avail 2024-02-10T09:20:15.656 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T09:20:15.656 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T09:20:15.656 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-02-10T09:20:15.656 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-02-10T09:20:15.656 INFO:teuthology.orchestra.run.smithi204.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134607 s, 779 MB/s 2024-02-10T09:20:15.656 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: cluster 2024-02-10T09:20:14.414422+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: cluster 2024-02-10T09:20:14.414483+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: audit 2024-02-10T09:20:14.599439+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: cluster 2024-02-10T09:20:14.599509+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e36: 5 total, 4 up, 4 in 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: audit 2024-02-10T09:20:14.600293+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: cluster 2024-02-10T09:20:15.599574+0000 mon.a (mon.0) 396 : cluster [INF] osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970] boot 2024-02-10T09:20:15.885 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: cluster 2024-02-10T09:20:15.599636+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-02-10T09:20:15.886 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:15 smithi204 bash[35809]: audit 2024-02-10T09:20:15.600667+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:15.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: cluster 2024-02-10T09:20:14.414422+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: cluster 2024-02-10T09:20:14.414483+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: audit 2024-02-10T09:20:14.599439+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: cluster 2024-02-10T09:20:14.599509+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e36: 5 total, 4 up, 4 in 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: audit 2024-02-10T09:20:14.600293+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: cluster 2024-02-10T09:20:15.599574+0000 mon.a (mon.0) 396 : cluster [INF] osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970] boot 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: cluster 2024-02-10T09:20:15.599636+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[35780]: audit 2024-02-10T09:20:15.600667+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:15.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: cluster 2024-02-10T09:20:14.414422+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: cluster 2024-02-10T09:20:14.414483+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: audit 2024-02-10T09:20:14.599439+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: cluster 2024-02-10T09:20:14.599509+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e36: 5 total, 4 up, 4 in 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: audit 2024-02-10T09:20:14.600293+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: cluster 2024-02-10T09:20:15.599574+0000 mon.a (mon.0) 396 : cluster [INF] osd.4 [v2:172.21.15.204:6800/1147358970,v1:172.21.15.204:6801/1147358970] boot 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: cluster 2024-02-10T09:20:15.599636+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-02-10T09:20:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:15 smithi153 bash[39117]: audit 2024-02-10T09:20:15.600667+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:20:16.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-02-10T09:20:16.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:16 smithi204 bash[35809]: cluster 2024-02-10T09:20:15.773871+0000 mgr.y (mgr.14140) 110 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.4 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:16.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:16 smithi204 bash[35809]: cluster 2024-02-10T09:20:16.600448+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-02-10T09:20:16.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:16 smithi153 bash[39117]: cluster 2024-02-10T09:20:15.773871+0000 mgr.y (mgr.14140) 110 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.4 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:16.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:16 smithi153 bash[39117]: cluster 2024-02-10T09:20:16.600448+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-02-10T09:20:16.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:16 smithi153 bash[35780]: cluster 2024-02-10T09:20:15.773871+0000 mgr.y (mgr.14140) 110 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.4 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:16.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:16 smithi153 bash[35780]: cluster 2024-02-10T09:20:16.600448+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-02-10T09:20:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:17 smithi204 bash[35809]: cluster 2024-02-10T09:20:17.774243+0000 mgr.y (mgr.14140) 111 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.5 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:18.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:17 smithi153 bash[35780]: cluster 2024-02-10T09:20:17.774243+0000 mgr.y (mgr.14140) 111 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.5 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:17 smithi153 bash[39117]: cluster 2024-02-10T09:20:17.774243+0000 mgr.y (mgr.14140) 111 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.5 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:19.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:19 smithi204 bash[35809]: audit 2024-02-10T09:20:19.050245+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:19.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:19 smithi204 bash[35809]: audit 2024-02-10T09:20:19.051007+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:19.434 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:19 smithi204 bash[35809]: audit 2024-02-10T09:20:19.052096+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:19 smithi153 bash[35780]: audit 2024-02-10T09:20:19.050245+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:19 smithi153 bash[35780]: audit 2024-02-10T09:20:19.051007+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:19 smithi153 bash[35780]: audit 2024-02-10T09:20:19.052096+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:19 smithi153 bash[39117]: audit 2024-02-10T09:20:19.050245+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:19 smithi153 bash[39117]: audit 2024-02-10T09:20:19.051007+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:19 smithi153 bash[39117]: audit 2024-02-10T09:20:19.052096+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:20.320 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:20 smithi204 bash[35809]: audit 2024-02-10T09:20:19.049209+0000 mgr.y (mgr.14140) 112 : audit [DBG] from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:20.320 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:20 smithi204 bash[35809]: cephadm 2024-02-10T09:20:19.049950+0000 mgr.y (mgr.14140) 113 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:20.320 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:20 smithi204 bash[35809]: audit 2024-02-10T09:20:19.151756+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:20.320 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:20 smithi204 bash[35809]: audit 2024-02-10T09:20:19.345043+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:20.320 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:20 smithi204 bash[35809]: audit 2024-02-10T09:20:19.348998+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:20.320 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:20 smithi204 bash[35809]: cluster 2024-02-10T09:20:19.774617+0000 mgr.y (mgr.14140) 114 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.5 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:20.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[35780]: audit 2024-02-10T09:20:19.049209+0000 mgr.y (mgr.14140) 112 : audit [DBG] from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:20.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[35780]: cephadm 2024-02-10T09:20:19.049950+0000 mgr.y (mgr.14140) 113 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:20.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[35780]: audit 2024-02-10T09:20:19.151756+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:20.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[39117]: audit 2024-02-10T09:20:19.049209+0000 mgr.y (mgr.14140) 112 : audit [DBG] from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:20.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[39117]: cephadm 2024-02-10T09:20:19.049950+0000 mgr.y (mgr.14140) 113 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:20.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[39117]: audit 2024-02-10T09:20:19.151756+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:20.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[35780]: audit 2024-02-10T09:20:19.345043+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:20.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[39117]: audit 2024-02-10T09:20:19.345043+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:20.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[35780]: audit 2024-02-10T09:20:19.348998+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:20.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[35780]: cluster 2024-02-10T09:20:19.774617+0000 mgr.y (mgr.14140) 114 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.5 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:20.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[39117]: audit 2024-02-10T09:20:19.348998+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}}, \"last_daemon_update\": \"2024-02-10T09:20:13.527263\", \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:20.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:20 smithi153 bash[39117]: cluster 2024-02-10T09:20:19.774617+0000 mgr.y (mgr.14140) 114 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.5 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: audit 2024-02-10T09:20:20.628366+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: audit 2024-02-10T09:20:20.631984+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: audit 2024-02-10T09:20:21.247482+0000 mon.b (mon.2) 8 : audit [INF] from='client.? 172.21.15.204:0/1044589130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]: dispatch 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: audit 2024-02-10T09:20:21.247865+0000 mon.a (mon.0) 408 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]: dispatch 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: audit 2024-02-10T09:20:21.253010+0000 mon.a (mon.0) 409 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]': finished 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: cluster 2024-02-10T09:20:21.253090+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e39: 6 total, 5 up, 5 in 2024-02-10T09:20:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[35780]: audit 2024-02-10T09:20:21.253247+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: audit 2024-02-10T09:20:20.628366+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: audit 2024-02-10T09:20:20.631984+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: audit 2024-02-10T09:20:21.247482+0000 mon.b (mon.2) 8 : audit [INF] from='client.? 172.21.15.204:0/1044589130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]: dispatch 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: audit 2024-02-10T09:20:21.247865+0000 mon.a (mon.0) 408 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]: dispatch 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: audit 2024-02-10T09:20:21.253010+0000 mon.a (mon.0) 409 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]': finished 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: cluster 2024-02-10T09:20:21.253090+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e39: 6 total, 5 up, 5 in 2024-02-10T09:20:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:21 smithi153 bash[39117]: audit 2024-02-10T09:20:21.253247+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: audit 2024-02-10T09:20:20.628366+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: audit 2024-02-10T09:20:20.631984+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: audit 2024-02-10T09:20:21.247482+0000 mon.b (mon.2) 8 : audit [INF] from='client.? 172.21.15.204:0/1044589130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]: dispatch 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: audit 2024-02-10T09:20:21.247865+0000 mon.a (mon.0) 408 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]: dispatch 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: audit 2024-02-10T09:20:21.253010+0000 mon.a (mon.0) 409 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ed1e885-ba58-4d66-a134-e63d325d4fcb"}]': finished 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: cluster 2024-02-10T09:20:21.253090+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e39: 6 total, 5 up, 5 in 2024-02-10T09:20:21.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:21 smithi204 bash[35809]: audit 2024-02-10T09:20:21.253247+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:22 smithi153 bash[35780]: cluster 2024-02-10T09:20:21.774959+0000 mgr.y (mgr.14140) 115 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:22 smithi153 bash[35780]: audit 2024-02-10T09:20:21.998376+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? 172.21.15.204:0/3566741637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:22 smithi153 bash[39117]: cluster 2024-02-10T09:20:21.774959+0000 mgr.y (mgr.14140) 115 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:22 smithi153 bash[39117]: audit 2024-02-10T09:20:21.998376+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? 172.21.15.204:0/3566741637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:22 smithi204 bash[35809]: cluster 2024-02-10T09:20:21.774959+0000 mgr.y (mgr.14140) 115 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:22 smithi204 bash[35809]: audit 2024-02-10T09:20:21.998376+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? 172.21.15.204:0/3566741637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:24.188 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:23 smithi204 bash[35809]: cluster 2024-02-10T09:20:23.775417+0000 mgr.y (mgr.14140) 116 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:23 smithi153 bash[35780]: cluster 2024-02-10T09:20:23.775417+0000 mgr.y (mgr.14140) 116 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:23 smithi153 bash[39117]: cluster 2024-02-10T09:20:23.775417+0000 mgr.y (mgr.14140) 116 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:24 smithi204 bash[35809]: audit 2024-02-10T09:20:24.415536+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:25.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:24 smithi153 bash[39117]: audit 2024-02-10T09:20:24.415536+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:24 smithi153 bash[35780]: audit 2024-02-10T09:20:24.415536+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:25.938 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:25 smithi204 bash[35809]: cluster 2024-02-10T09:20:25.775918+0000 mgr.y (mgr.14140) 117 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:25 smithi153 bash[39117]: cluster 2024-02-10T09:20:25.775918+0000 mgr.y (mgr.14140) 117 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:25 smithi153 bash[35780]: cluster 2024-02-10T09:20:25.775918+0000 mgr.y (mgr.14140) 117 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T09:20:27.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:27 smithi204 bash[35809]: audit 2024-02-10T09:20:26.993724+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:20:27.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:27 smithi204 bash[35809]: audit 2024-02-10T09:20:26.994511+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:27.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:27 smithi204 bash[35809]: audit 2024-02-10T09:20:26.995351+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.5", "key": "container_image"}]: dispatch 2024-02-10T09:20:27.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:27 smithi153 bash[35780]: audit 2024-02-10T09:20:26.993724+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:20:27.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:27 smithi153 bash[35780]: audit 2024-02-10T09:20:26.994511+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:27.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:27 smithi153 bash[35780]: audit 2024-02-10T09:20:26.995351+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.5", "key": "container_image"}]: dispatch 2024-02-10T09:20:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:27 smithi153 bash[39117]: audit 2024-02-10T09:20:26.993724+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:20:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:27 smithi153 bash[39117]: audit 2024-02-10T09:20:26.994511+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:27 smithi153 bash[39117]: audit 2024-02-10T09:20:26.995351+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.5", "key": "container_image"}]: dispatch 2024-02-10T09:20:28.529 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-02-10T09:20:28.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:28 smithi204 bash[35809]: cephadm 2024-02-10T09:20:26.995073+0000 mgr.y (mgr.14140) 118 : cephadm [INF] Deploying daemon osd.5 on smithi204 2024-02-10T09:20:28.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:28 smithi204 bash[35809]: cluster 2024-02-10T09:20:27.776233+0000 mgr.y (mgr.14140) 119 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:28.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:28 smithi204 bash[35809]: audit 2024-02-10T09:20:28.523066+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:28.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:28 smithi204 bash[35809]: audit 2024-02-10T09:20:28.523517+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:28.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:28 smithi204 bash[35809]: audit 2024-02-10T09:20:28.526548+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:28.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[35780]: cephadm 2024-02-10T09:20:26.995073+0000 mgr.y (mgr.14140) 118 : cephadm [INF] Deploying daemon osd.5 on smithi204 2024-02-10T09:20:28.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[35780]: cluster 2024-02-10T09:20:27.776233+0000 mgr.y (mgr.14140) 119 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[39117]: cephadm 2024-02-10T09:20:26.995073+0000 mgr.y (mgr.14140) 118 : cephadm [INF] Deploying daemon osd.5 on smithi204 2024-02-10T09:20:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[39117]: cluster 2024-02-10T09:20:27.776233+0000 mgr.y (mgr.14140) 119 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[39117]: audit 2024-02-10T09:20:28.523066+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[39117]: audit 2024-02-10T09:20:28.523517+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[39117]: audit 2024-02-10T09:20:28.526548+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:28.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[35780]: audit 2024-02-10T09:20:28.523066+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:28.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[35780]: audit 2024-02-10T09:20:28.523517+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:28.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:28 smithi153 bash[35780]: audit 2024-02-10T09:20:28.526548+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527043\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527138\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:13.527173\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_device_update\": \"2024-02-10T09:20:19.344470\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:30.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:29 smithi204 bash[35809]: cluster 2024-02-10T09:20:29.776539+0000 mgr.y (mgr.14140) 120 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:30.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:29 smithi153 bash[35780]: cluster 2024-02-10T09:20:29.776539+0000 mgr.y (mgr.14140) 120 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:29 smithi153 bash[39117]: cluster 2024-02-10T09:20:29.776539+0000 mgr.y (mgr.14140) 120 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:30.715 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T09:20:30.717 INFO:tasks.cephadm:Deploying osd.6 on smithi204 with /dev/vg_nvme/lv_2... 2024-02-10T09:20:30.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T09:20:30.748 INFO:journalctl@ceph.osd.5.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:20:32.081 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:20:31 smithi204 bash[39725]: debug 2024-02-10T09:20:31.783+0000 7f6642ef6ec0 -1 Falling back to public interface 2024-02-10T09:20:32.439 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:20:32 smithi204 bash[39725]: debug 2024-02-10T09:20:32.326+0000 7f6642ef6ec0 -1 osd.5 0 log_to_monitors {default=true} 2024-02-10T09:20:32.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:32 smithi204 bash[35809]: cluster 2024-02-10T09:20:31.776863+0000 mgr.y (mgr.14140) 121 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:32.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:32 smithi204 bash[35809]: audit 2024-02-10T09:20:32.329929+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:20:32.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:32 smithi204 bash[35809]: audit 2024-02-10T09:20:32.330290+0000 mon.a (mon.0) 419 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:20:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:32 smithi153 bash[35780]: cluster 2024-02-10T09:20:31.776863+0000 mgr.y (mgr.14140) 121 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:32 smithi153 bash[35780]: audit 2024-02-10T09:20:32.329929+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:20:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:32 smithi153 bash[35780]: audit 2024-02-10T09:20:32.330290+0000 mon.a (mon.0) 419 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:20:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:32 smithi153 bash[39117]: cluster 2024-02-10T09:20:31.776863+0000 mgr.y (mgr.14140) 121 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:32 smithi153 bash[39117]: audit 2024-02-10T09:20:32.329929+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:20:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:32 smithi153 bash[39117]: audit 2024-02-10T09:20:32.330290+0000 mon.a (mon.0) 419 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:20:32.970 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T09:20:32.970 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T09:20:32.970 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-02-10T09:20:32.970 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-02-10T09:20:32.970 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126644 s, 828 MB/s 2024-02-10T09:20:32.970 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-02-10T09:20:33.831 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[39725]: debug 2024-02-10T09:20:33.590+0000 7f6627ca3700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T09:20:33.831 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[39725]: debug 2024-02-10T09:20:33.599+0000 7f6632ebf700 -1 osd.5 41 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.586784+0000 mon.a (mon.0) 420 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: cluster 2024-02-10T09:20:32.586864+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e40: 6 total, 5 up, 5 in 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.587008+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.587290+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.587548+0000 mon.a (mon.0) 423 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.735054+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.738100+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: audit 2024-02-10T09:20:32.738815+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: cluster 2024-02-10T09:20:33.299202+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:33.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:33 smithi204 bash[35809]: cluster 2024-02-10T09:20:33.299240+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.586784+0000 mon.a (mon.0) 420 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T09:20:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: cluster 2024-02-10T09:20:32.586864+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e40: 6 total, 5 up, 5 in 2024-02-10T09:20:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.587008+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.587290+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.587548+0000 mon.a (mon.0) 423 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:33.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.586784+0000 mon.a (mon.0) 420 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: cluster 2024-02-10T09:20:32.586864+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e40: 6 total, 5 up, 5 in 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.587008+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.587290+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.587548+0000 mon.a (mon.0) 423 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.735054+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.738100+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: audit 2024-02-10T09:20:32.738815+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: cluster 2024-02-10T09:20:33.299202+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:33.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[39117]: cluster 2024-02-10T09:20:33.299240+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:33.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.735054+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:33.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.738100+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:33.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: audit 2024-02-10T09:20:32.738815+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:33.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: cluster 2024-02-10T09:20:33.299202+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:33.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:33 smithi153 bash[35780]: cluster 2024-02-10T09:20:33.299240+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:34.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:34 smithi204 bash[35809]: audit 2024-02-10T09:20:33.588871+0000 mon.a (mon.0) 427 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:34.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:34 smithi204 bash[35809]: cluster 2024-02-10T09:20:33.588932+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e41: 6 total, 5 up, 5 in 2024-02-10T09:20:34.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:34 smithi204 bash[35809]: audit 2024-02-10T09:20:33.589787+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:34.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:34 smithi204 bash[35809]: cluster 2024-02-10T09:20:33.777255+0000 mgr.y (mgr.14140) 122 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:34.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[35780]: audit 2024-02-10T09:20:33.588871+0000 mon.a (mon.0) 427 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[35780]: cluster 2024-02-10T09:20:33.588932+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e41: 6 total, 5 up, 5 in 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[35780]: audit 2024-02-10T09:20:33.589787+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[35780]: cluster 2024-02-10T09:20:33.777255+0000 mgr.y (mgr.14140) 122 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[39117]: audit 2024-02-10T09:20:33.588871+0000 mon.a (mon.0) 427 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[39117]: cluster 2024-02-10T09:20:33.588932+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e41: 6 total, 5 up, 5 in 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[39117]: audit 2024-02-10T09:20:33.589787+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:34 smithi153 bash[39117]: cluster 2024-02-10T09:20:33.777255+0000 mgr.y (mgr.14140) 122 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.6 MiB used, 442 GiB / 447 GiB avail 2024-02-10T09:20:35.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:35 smithi204 bash[35809]: cluster 2024-02-10T09:20:34.590480+0000 mon.a (mon.0) 430 : cluster [INF] osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539] boot 2024-02-10T09:20:35.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:35 smithi204 bash[35809]: cluster 2024-02-10T09:20:34.590531+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-02-10T09:20:35.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:35 smithi204 bash[35809]: audit 2024-02-10T09:20:34.590823+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:35 smithi153 bash[35780]: cluster 2024-02-10T09:20:34.590480+0000 mon.a (mon.0) 430 : cluster [INF] osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539] boot 2024-02-10T09:20:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:35 smithi153 bash[35780]: cluster 2024-02-10T09:20:34.590531+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-02-10T09:20:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:35 smithi153 bash[35780]: audit 2024-02-10T09:20:34.590823+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:35 smithi153 bash[39117]: cluster 2024-02-10T09:20:34.590480+0000 mon.a (mon.0) 430 : cluster [INF] osd.5 [v2:172.21.15.204:6808/4030840539,v1:172.21.15.204:6809/4030840539] boot 2024-02-10T09:20:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:35 smithi153 bash[39117]: cluster 2024-02-10T09:20:34.590531+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-02-10T09:20:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:35 smithi153 bash[39117]: audit 2024-02-10T09:20:34.590823+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:20:36.717 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:36 smithi204 bash[35809]: cluster 2024-02-10T09:20:35.605344+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-02-10T09:20:36.718 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:36 smithi204 bash[35809]: cluster 2024-02-10T09:20:35.777758+0000 mgr.y (mgr.14140) 123 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.0 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:36.718 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:36 smithi204 bash[35809]: audit 2024-02-10T09:20:36.014962+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:36.718 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:36 smithi204 bash[35809]: audit 2024-02-10T09:20:36.197805+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:36.718 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:36 smithi204 bash[35809]: audit 2024-02-10T09:20:36.200985+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:36.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[35780]: cluster 2024-02-10T09:20:35.605344+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-02-10T09:20:36.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[35780]: cluster 2024-02-10T09:20:35.777758+0000 mgr.y (mgr.14140) 123 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.0 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:36.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[35780]: audit 2024-02-10T09:20:36.014962+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[39117]: cluster 2024-02-10T09:20:35.605344+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-02-10T09:20:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[39117]: cluster 2024-02-10T09:20:35.777758+0000 mgr.y (mgr.14140) 123 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.0 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[39117]: audit 2024-02-10T09:20:36.014962+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[39117]: audit 2024-02-10T09:20:36.197805+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[39117]: audit 2024-02-10T09:20:36.200985+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[35780]: audit 2024-02-10T09:20:36.197805+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:36.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:36 smithi153 bash[35780]: audit 2024-02-10T09:20:36.200985+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}}, \"last_daemon_update\": \"2024-02-10T09:20:32.734439\", \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:37.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:37 smithi153 bash[39117]: audit 2024-02-10T09:20:37.362703+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:37.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:37 smithi153 bash[39117]: audit 2024-02-10T09:20:37.363449+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:37.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:37 smithi153 bash[39117]: audit 2024-02-10T09:20:37.364583+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:37.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:37 smithi153 bash[35780]: audit 2024-02-10T09:20:37.362703+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:37.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:37 smithi153 bash[35780]: audit 2024-02-10T09:20:37.363449+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:37.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:37 smithi153 bash[35780]: audit 2024-02-10T09:20:37.364583+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:37.987 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:37 smithi204 bash[35809]: audit 2024-02-10T09:20:37.362703+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:37.987 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:37 smithi204 bash[35809]: audit 2024-02-10T09:20:37.363449+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:37.987 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:37 smithi204 bash[35809]: audit 2024-02-10T09:20:37.364583+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:38.652 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:38 smithi204 bash[35809]: audit 2024-02-10T09:20:37.361628+0000 mgr.y (mgr.14140) 124 : audit [DBG] from='client.34253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:38.652 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:38 smithi204 bash[35809]: cephadm 2024-02-10T09:20:37.362391+0000 mgr.y (mgr.14140) 125 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:38.652 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:38 smithi204 bash[35809]: cluster 2024-02-10T09:20:37.778223+0000 mgr.y (mgr.14140) 126 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.1 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:38.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:38 smithi153 bash[35780]: audit 2024-02-10T09:20:37.361628+0000 mgr.y (mgr.14140) 124 : audit [DBG] from='client.34253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:38.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:38 smithi153 bash[35780]: cephadm 2024-02-10T09:20:37.362391+0000 mgr.y (mgr.14140) 125 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:38.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:38 smithi153 bash[35780]: cluster 2024-02-10T09:20:37.778223+0000 mgr.y (mgr.14140) 126 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.1 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:38.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:38 smithi153 bash[39117]: audit 2024-02-10T09:20:37.361628+0000 mgr.y (mgr.14140) 124 : audit [DBG] from='client.34253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:38.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:38 smithi153 bash[39117]: cephadm 2024-02-10T09:20:37.362391+0000 mgr.y (mgr.14140) 125 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:38.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:38 smithi153 bash[39117]: cluster 2024-02-10T09:20:37.778223+0000 mgr.y (mgr.14140) 126 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.1 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[35780]: audit 2024-02-10T09:20:39.555656+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.204:0/4130820838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]: dispatch 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[35780]: audit 2024-02-10T09:20:39.556124+0000 mon.a (mon.0) 440 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]: dispatch 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[35780]: audit 2024-02-10T09:20:39.563141+0000 mon.a (mon.0) 441 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]': finished 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[35780]: cluster 2024-02-10T09:20:39.563198+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e44: 7 total, 6 up, 6 in 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[35780]: audit 2024-02-10T09:20:39.563326+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[39117]: audit 2024-02-10T09:20:39.555656+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.204:0/4130820838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]: dispatch 2024-02-10T09:20:39.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[39117]: audit 2024-02-10T09:20:39.556124+0000 mon.a (mon.0) 440 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]: dispatch 2024-02-10T09:20:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[39117]: audit 2024-02-10T09:20:39.563141+0000 mon.a (mon.0) 441 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]': finished 2024-02-10T09:20:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[39117]: cluster 2024-02-10T09:20:39.563198+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e44: 7 total, 6 up, 6 in 2024-02-10T09:20:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:39 smithi153 bash[39117]: audit 2024-02-10T09:20:39.563326+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:39 smithi204 bash[35809]: audit 2024-02-10T09:20:39.555656+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.204:0/4130820838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]: dispatch 2024-02-10T09:20:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:39 smithi204 bash[35809]: audit 2024-02-10T09:20:39.556124+0000 mon.a (mon.0) 440 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]: dispatch 2024-02-10T09:20:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:39 smithi204 bash[35809]: audit 2024-02-10T09:20:39.563141+0000 mon.a (mon.0) 441 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f3e1b43-d7dd-44b2-be39-d240e518a51c"}]': finished 2024-02-10T09:20:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:39 smithi204 bash[35809]: cluster 2024-02-10T09:20:39.563198+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e44: 7 total, 6 up, 6 in 2024-02-10T09:20:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:39 smithi204 bash[35809]: audit 2024-02-10T09:20:39.563326+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:40.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:40 smithi153 bash[35780]: cluster 2024-02-10T09:20:39.778689+0000 mgr.y (mgr.14140) 127 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.2 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:40.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:40 smithi153 bash[35780]: audit 2024-02-10T09:20:40.292376+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.204:0/1257198868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:40.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:40 smithi153 bash[39117]: cluster 2024-02-10T09:20:39.778689+0000 mgr.y (mgr.14140) 127 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.2 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:40.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:40 smithi153 bash[39117]: audit 2024-02-10T09:20:40.292376+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.204:0/1257198868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:40 smithi204 bash[35809]: cluster 2024-02-10T09:20:39.778689+0000 mgr.y (mgr.14140) 127 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.2 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:40 smithi204 bash[35809]: audit 2024-02-10T09:20:40.292376+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.204:0/1257198868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:42.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:42 smithi204 bash[35809]: cluster 2024-02-10T09:20:41.779038+0000 mgr.y (mgr.14140) 128 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.2 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:42.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:42 smithi153 bash[35780]: cluster 2024-02-10T09:20:41.779038+0000 mgr.y (mgr.14140) 128 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.2 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:42.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:42 smithi153 bash[39117]: cluster 2024-02-10T09:20:41.779038+0000 mgr.y (mgr.14140) 128 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.2 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:43.784 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:43 smithi204 bash[35809]: audit 2024-02-10T09:20:42.782264+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:43.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:43 smithi153 bash[35780]: audit 2024-02-10T09:20:42.782264+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:43.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:43 smithi153 bash[39117]: audit 2024-02-10T09:20:42.782264+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:44.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:44 smithi153 bash[35780]: cluster 2024-02-10T09:20:43.779448+0000 mgr.y (mgr.14140) 129 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:44.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:44 smithi153 bash[39117]: cluster 2024-02-10T09:20:43.779448+0000 mgr.y (mgr.14140) 129 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:45.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:44 smithi204 bash[35809]: cluster 2024-02-10T09:20:43.779448+0000 mgr.y (mgr.14140) 129 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:45.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:45 smithi204 bash[35809]: audit 2024-02-10T09:20:45.393631+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:20:45.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:45 smithi204 bash[35809]: audit 2024-02-10T09:20:45.394439+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:45.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:45 smithi204 bash[35809]: audit 2024-02-10T09:20:45.395249+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.6", "key": "container_image"}]: dispatch 2024-02-10T09:20:45.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:45 smithi153 bash[35780]: audit 2024-02-10T09:20:45.393631+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:20:45.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:45 smithi153 bash[35780]: audit 2024-02-10T09:20:45.394439+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:45.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:45 smithi153 bash[35780]: audit 2024-02-10T09:20:45.395249+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.6", "key": "container_image"}]: dispatch 2024-02-10T09:20:45.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:45 smithi153 bash[39117]: audit 2024-02-10T09:20:45.393631+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:20:45.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:45 smithi153 bash[39117]: audit 2024-02-10T09:20:45.394439+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:45.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:45 smithi153 bash[39117]: audit 2024-02-10T09:20:45.395249+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.6", "key": "container_image"}]: dispatch 2024-02-10T09:20:46.712 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:46 smithi204 bash[35809]: cephadm 2024-02-10T09:20:45.394956+0000 mgr.y (mgr.14140) 130 : cephadm [INF] Deploying daemon osd.6 on smithi204 2024-02-10T09:20:46.712 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:46 smithi204 bash[35809]: cluster 2024-02-10T09:20:45.779817+0000 mgr.y (mgr.14140) 131 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:46.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:46 smithi153 bash[35780]: cephadm 2024-02-10T09:20:45.394956+0000 mgr.y (mgr.14140) 130 : cephadm [INF] Deploying daemon osd.6 on smithi204 2024-02-10T09:20:46.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:46 smithi153 bash[35780]: cluster 2024-02-10T09:20:45.779817+0000 mgr.y (mgr.14140) 131 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:46.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:46 smithi153 bash[39117]: cephadm 2024-02-10T09:20:45.394956+0000 mgr.y (mgr.14140) 130 : cephadm [INF] Deploying daemon osd.6 on smithi204 2024-02-10T09:20:46.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:46 smithi153 bash[39117]: cluster 2024-02-10T09:20:45.779817+0000 mgr.y (mgr.14140) 131 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:47.039 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 6 on host 'smithi204' 2024-02-10T09:20:47.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:47 smithi153 bash[35780]: audit 2024-02-10T09:20:47.032315+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:47.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:47 smithi153 bash[35780]: audit 2024-02-10T09:20:47.032745+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:47.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:47 smithi153 bash[35780]: audit 2024-02-10T09:20:47.035492+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:47.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:47 smithi153 bash[39117]: audit 2024-02-10T09:20:47.032315+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:47.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:47 smithi153 bash[39117]: audit 2024-02-10T09:20:47.032745+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:47.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:47 smithi153 bash[39117]: audit 2024-02-10T09:20:47.035492+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:47.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:47 smithi204 bash[35809]: audit 2024-02-10T09:20:47.032315+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:47.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:47 smithi204 bash[35809]: audit 2024-02-10T09:20:47.032745+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:47.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:47 smithi204 bash[35809]: audit 2024-02-10T09:20:47.035492+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734199\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734291\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734326\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:32.734357\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_device_update\": \"2024-02-10T09:20:36.197155\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:48 smithi204 bash[35809]: cluster 2024-02-10T09:20:47.780259+0000 mgr.y (mgr.14140) 132 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:48.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:48 smithi153 bash[35780]: cluster 2024-02-10T09:20:47.780259+0000 mgr.y (mgr.14140) 132 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:48.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:48 smithi153 bash[39117]: cluster 2024-02-10T09:20:47.780259+0000 mgr.y (mgr.14140) 132 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:49.060 DEBUG:teuthology.orchestra.run.smithi204:osd.6> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T09:20:49.062 INFO:tasks.cephadm:Deploying osd.7 on smithi204 with /dev/vg_nvme/lv_1... 2024-02-10T09:20:49.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T09:20:49.094 INFO:journalctl@ceph.osd.6.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:20:49.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:49 smithi204 bash[35809]: cluster 2024-02-10T09:20:49.780676+0000 mgr.y (mgr.14140) 133 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:49 smithi153 bash[35780]: cluster 2024-02-10T09:20:49.780676+0000 mgr.y (mgr.14140) 133 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:49 smithi153 bash[39117]: cluster 2024-02-10T09:20:49.780676+0000 mgr.y (mgr.14140) 133 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:50.831 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:20:50 smithi204 bash[41390]: debug 2024-02-10T09:20:50.578+0000 7ff47cf7aec0 -1 Falling back to public interface 2024-02-10T09:20:51.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:50 smithi204 bash[35809]: audit 2024-02-10T09:20:50.163434+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:51.138 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:50 smithi204 bash[35809]: audit 2024-02-10T09:20:50.168723+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:51.138 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:50 smithi204 bash[35809]: audit 2024-02-10T09:20:50.169412+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:51.138 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:50 smithi204 bash[35809]: audit 2024-02-10T09:20:50.633563+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:20:51.138 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:50 smithi204 bash[35809]: audit 2024-02-10T09:20:50.636717+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:20:51.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[39117]: audit 2024-02-10T09:20:50.163434+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[35780]: audit 2024-02-10T09:20:50.163434+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[35780]: audit 2024-02-10T09:20:50.168723+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[35780]: audit 2024-02-10T09:20:50.169412+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[35780]: audit 2024-02-10T09:20:50.633563+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:20:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[35780]: audit 2024-02-10T09:20:50.636717+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:20:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[39117]: audit 2024-02-10T09:20:50.168723+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:51.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[39117]: audit 2024-02-10T09:20:50.169412+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:51.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[39117]: audit 2024-02-10T09:20:50.633563+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:20:51.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:50 smithi153 bash[39117]: audit 2024-02-10T09:20:50.636717+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:20:51.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:20:51 smithi204 bash[41390]: debug 2024-02-10T09:20:51.136+0000 7ff47cf7aec0 -1 osd.6 0 log_to_monitors {default=true} 2024-02-10T09:20:51.672 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T09:20:51.673 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T09:20:51.673 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-02-10T09:20:51.673 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-02-10T09:20:51.673 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110464 s, 949 MB/s 2024-02-10T09:20:51.673 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-02-10T09:20:52.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:51 smithi204 bash[35809]: audit 2024-02-10T09:20:51.140283+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:20:52.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:51 smithi204 bash[35809]: audit 2024-02-10T09:20:51.140773+0000 mon.a (mon.0) 456 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:20:52.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:51 smithi204 bash[35809]: cluster 2024-02-10T09:20:51.780977+0000 mgr.y (mgr.14140) 134 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:51 smithi153 bash[35780]: audit 2024-02-10T09:20:51.140283+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:20:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:51 smithi153 bash[35780]: audit 2024-02-10T09:20:51.140773+0000 mon.a (mon.0) 456 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:20:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:51 smithi153 bash[35780]: cluster 2024-02-10T09:20:51.780977+0000 mgr.y (mgr.14140) 134 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:51 smithi153 bash[39117]: audit 2024-02-10T09:20:51.140283+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:20:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:51 smithi153 bash[39117]: audit 2024-02-10T09:20:51.140773+0000 mon.a (mon.0) 456 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:20:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:51 smithi153 bash[39117]: cluster 2024-02-10T09:20:51.780977+0000 mgr.y (mgr.14140) 134 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:52.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: audit 2024-02-10T09:20:51.888261+0000 mon.a (mon.0) 457 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: cluster 2024-02-10T09:20:51.888319+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e45: 7 total, 6 up, 6 in 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: audit 2024-02-10T09:20:51.888466+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: audit 2024-02-10T09:20:51.890733+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: audit 2024-02-10T09:20:51.891003+0000 mon.a (mon.0) 460 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: cluster 2024-02-10T09:20:52.131272+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[39117]: cluster 2024-02-10T09:20:52.131336+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: audit 2024-02-10T09:20:51.888261+0000 mon.a (mon.0) 457 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: cluster 2024-02-10T09:20:51.888319+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e45: 7 total, 6 up, 6 in 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: audit 2024-02-10T09:20:51.888466+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: audit 2024-02-10T09:20:51.890733+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: audit 2024-02-10T09:20:51.891003+0000 mon.a (mon.0) 460 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: cluster 2024-02-10T09:20:52.131272+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:52 smithi153 bash[35780]: cluster 2024-02-10T09:20:52.131336+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:53.306 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[41390]: debug 2024-02-10T09:20:52.898+0000 7ff465d2f700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T09:20:53.306 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[41390]: debug 2024-02-10T09:20:52.904+0000 7ff470f4b700 -1 osd.6 46 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: audit 2024-02-10T09:20:51.888261+0000 mon.a (mon.0) 457 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: cluster 2024-02-10T09:20:51.888319+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e45: 7 total, 6 up, 6 in 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: audit 2024-02-10T09:20:51.888466+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: audit 2024-02-10T09:20:51.890733+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: audit 2024-02-10T09:20:51.891003+0000 mon.a (mon.0) 460 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: cluster 2024-02-10T09:20:52.131272+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:20:53.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:52 smithi204 bash[35809]: cluster 2024-02-10T09:20:52.131336+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:20:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:53 smithi204 bash[35809]: audit 2024-02-10T09:20:52.890106+0000 mon.a (mon.0) 461 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:53 smithi204 bash[35809]: cluster 2024-02-10T09:20:52.890171+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e46: 7 total, 6 up, 6 in 2024-02-10T09:20:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:53 smithi204 bash[35809]: audit 2024-02-10T09:20:52.890714+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:53 smithi204 bash[35809]: cluster 2024-02-10T09:20:53.781251+0000 mgr.y (mgr.14140) 135 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[39117]: audit 2024-02-10T09:20:52.890106+0000 mon.a (mon.0) 461 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[39117]: cluster 2024-02-10T09:20:52.890171+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e46: 7 total, 6 up, 6 in 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[39117]: audit 2024-02-10T09:20:52.890714+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[39117]: cluster 2024-02-10T09:20:53.781251+0000 mgr.y (mgr.14140) 135 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[35780]: audit 2024-02-10T09:20:52.890106+0000 mon.a (mon.0) 461 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[35780]: cluster 2024-02-10T09:20:52.890171+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e46: 7 total, 6 up, 6 in 2024-02-10T09:20:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[35780]: audit 2024-02-10T09:20:52.890714+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:54.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:53 smithi153 bash[35780]: cluster 2024-02-10T09:20:53.781251+0000 mgr.y (mgr.14140) 135 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 MiB used, 530 GiB / 536 GiB avail 2024-02-10T09:20:55.081 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:20:54 smithi204 bash[39725]: debug 2024-02-10T09:20:54.905+0000 7f6618e8b700 -1 osd.5 48 build_incremental_map_msg missing incremental map 48 2024-02-10T09:20:55.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:54 smithi204 bash[35809]: cluster 2024-02-10T09:20:53.905448+0000 mon.a (mon.0) 464 : cluster [INF] osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292] boot 2024-02-10T09:20:55.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:54 smithi204 bash[35809]: cluster 2024-02-10T09:20:53.905500+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-10T09:20:55.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:54 smithi204 bash[35809]: audit 2024-02-10T09:20:53.905900+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:55.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:54 smithi153 bash[35780]: cluster 2024-02-10T09:20:53.905448+0000 mon.a (mon.0) 464 : cluster [INF] osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292] boot 2024-02-10T09:20:55.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:54 smithi153 bash[35780]: cluster 2024-02-10T09:20:53.905500+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-10T09:20:55.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:54 smithi153 bash[35780]: audit 2024-02-10T09:20:53.905900+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:54 smithi153 bash[39117]: cluster 2024-02-10T09:20:53.905448+0000 mon.a (mon.0) 464 : cluster [INF] osd.6 [v2:172.21.15.204:6816/2623910292,v1:172.21.15.204:6817/2623910292] boot 2024-02-10T09:20:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:54 smithi153 bash[39117]: cluster 2024-02-10T09:20:53.905500+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-10T09:20:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:54 smithi153 bash[39117]: audit 2024-02-10T09:20:53.905900+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:20:56.163 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: cluster 2024-02-10T09:20:54.906376+0000 mon.a (mon.0) 467 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.073567+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.265393+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.269652+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.766370+0000 mgr.y (mgr.14140) 136 : audit [DBG] from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: cephadm 2024-02-10T09:20:55.767089+0000 mgr.y (mgr.14140) 137 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.767417+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.768193+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:56.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: audit 2024-02-10T09:20:55.769432+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:56.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:55 smithi204 bash[35809]: cluster 2024-02-10T09:20:55.781532+0000 mgr.y (mgr.14140) 138 : cluster [DBG] pgmap v144: 1 pgs: 1 peering; 0 B data, 2.8 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:20:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: cluster 2024-02-10T09:20:54.906376+0000 mon.a (mon.0) 467 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-10T09:20:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.073567+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: cluster 2024-02-10T09:20:54.906376+0000 mon.a (mon.0) 467 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-10T09:20:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.073567+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:20:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.265393+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:56.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.265393+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:20:56.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.269652+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:56.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.766370+0000 mgr.y (mgr.14140) 136 : audit [DBG] from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:56.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.269652+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}}, \"last_daemon_update\": \"2024-02-10T09:20:50.162690\", \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.766370+0000 mgr.y (mgr.14140) 136 : audit [DBG] from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: cephadm 2024-02-10T09:20:55.767089+0000 mgr.y (mgr.14140) 137 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.767417+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.768193+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: audit 2024-02-10T09:20:55.769432+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[39117]: cluster 2024-02-10T09:20:55.781532+0000 mgr.y (mgr.14140) 138 : cluster [DBG] pgmap v144: 1 pgs: 1 peering; 0 B data, 2.8 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: cephadm 2024-02-10T09:20:55.767089+0000 mgr.y (mgr.14140) 137 : cephadm [INF] Applying osd on host smithi204... 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.767417+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T09:20:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.768193+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:20:56.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: audit 2024-02-10T09:20:55.769432+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:20:56.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:55 smithi153 bash[35780]: cluster 2024-02-10T09:20:55.781532+0000 mgr.y (mgr.14140) 138 : cluster [DBG] pgmap v144: 1 pgs: 1 peering; 0 B data, 2.8 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:20:57.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:56 smithi204 bash[35809]: cluster 2024-02-10T09:20:56.266830+0000 mon.a (mon.0) 474 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:20:57.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:56 smithi153 bash[35780]: cluster 2024-02-10T09:20:56.266830+0000 mon.a (mon.0) 474 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:20:57.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:56 smithi153 bash[39117]: cluster 2024-02-10T09:20:56.266830+0000 mon.a (mon.0) 474 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:20:58.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:57 smithi153 bash[35780]: cluster 2024-02-10T09:20:57.781828+0000 mgr.y (mgr.14140) 139 : cluster [DBG] pgmap v145: 1 pgs: 1 peering; 0 B data, 2.9 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:20:58.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:57 smithi153 bash[39117]: cluster 2024-02-10T09:20:57.781828+0000 mgr.y (mgr.14140) 139 : cluster [DBG] pgmap v145: 1 pgs: 1 peering; 0 B data, 2.9 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:20:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:57 smithi204 bash[35809]: cluster 2024-02-10T09:20:57.781828+0000 mgr.y (mgr.14140) 139 : cluster [DBG] pgmap v145: 1 pgs: 1 peering; 0 B data, 2.9 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[35780]: audit 2024-02-10T09:20:57.972210+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.204:0/2614284062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]: dispatch 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[35780]: audit 2024-02-10T09:20:57.972762+0000 mon.a (mon.0) 475 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]: dispatch 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[35780]: audit 2024-02-10T09:20:57.978431+0000 mon.a (mon.0) 476 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]': finished 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[35780]: cluster 2024-02-10T09:20:57.978476+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e49: 8 total, 7 up, 7 in 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[35780]: audit 2024-02-10T09:20:57.978618+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[35780]: audit 2024-02-10T09:20:58.716496+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.204:0/3642165012' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[39117]: audit 2024-02-10T09:20:57.972210+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.204:0/2614284062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]: dispatch 2024-02-10T09:20:59.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[39117]: audit 2024-02-10T09:20:57.972762+0000 mon.a (mon.0) 475 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]: dispatch 2024-02-10T09:20:59.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[39117]: audit 2024-02-10T09:20:57.978431+0000 mon.a (mon.0) 476 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]': finished 2024-02-10T09:20:59.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[39117]: cluster 2024-02-10T09:20:57.978476+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e49: 8 total, 7 up, 7 in 2024-02-10T09:20:59.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[39117]: audit 2024-02-10T09:20:57.978618+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:20:59.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:58 smithi153 bash[39117]: audit 2024-02-10T09:20:58.716496+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.204:0/3642165012' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:20:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:58 smithi204 bash[35809]: audit 2024-02-10T09:20:57.972210+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.204:0/2614284062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]: dispatch 2024-02-10T09:20:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:58 smithi204 bash[35809]: audit 2024-02-10T09:20:57.972762+0000 mon.a (mon.0) 475 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]: dispatch 2024-02-10T09:20:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:58 smithi204 bash[35809]: audit 2024-02-10T09:20:57.978431+0000 mon.a (mon.0) 476 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d38f33ce-8c0b-4235-be4f-4169e07e4cde"}]': finished 2024-02-10T09:20:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:58 smithi204 bash[35809]: cluster 2024-02-10T09:20:57.978476+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e49: 8 total, 7 up, 7 in 2024-02-10T09:20:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:58 smithi204 bash[35809]: audit 2024-02-10T09:20:57.978618+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:20:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:58 smithi204 bash[35809]: audit 2024-02-10T09:20:58.716496+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.204:0/3642165012' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T09:21:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:20:59 smithi153 bash[39117]: cluster 2024-02-10T09:20:59.782122+0000 mgr.y (mgr.14140) 140 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 3.0 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:20:59 smithi153 bash[35780]: cluster 2024-02-10T09:20:59.782122+0000 mgr.y (mgr.14140) 140 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 3.0 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:20:59 smithi204 bash[35809]: cluster 2024-02-10T09:20:59.782122+0000 mgr.y (mgr.14140) 140 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 3.0 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:01 smithi204 bash[35809]: audit 2024-02-10T09:21:01.294968+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:21:01.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:01 smithi153 bash[35780]: audit 2024-02-10T09:21:01.294968+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:21:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:01 smithi153 bash[39117]: audit 2024-02-10T09:21:01.294968+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:21:02.695 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:02 smithi204 bash[35809]: cluster 2024-02-10T09:21:01.782481+0000 mgr.y (mgr.14140) 141 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:02 smithi153 bash[35780]: cluster 2024-02-10T09:21:01.782481+0000 mgr.y (mgr.14140) 141 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:02 smithi153 bash[39117]: cluster 2024-02-10T09:21:01.782481+0000 mgr.y (mgr.14140) 141 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:03.779 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:03 smithi204 bash[35809]: cluster 2024-02-10T09:21:02.584283+0000 mon.a (mon.0) 480 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:21:03.779 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:03 smithi204 bash[35809]: cluster 2024-02-10T09:21:02.584313+0000 mon.a (mon.0) 481 : cluster [INF] Cluster is now healthy 2024-02-10T09:21:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:03 smithi153 bash[35780]: cluster 2024-02-10T09:21:02.584283+0000 mon.a (mon.0) 480 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:21:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:03 smithi153 bash[35780]: cluster 2024-02-10T09:21:02.584313+0000 mon.a (mon.0) 481 : cluster [INF] Cluster is now healthy 2024-02-10T09:21:03.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:03 smithi153 bash[39117]: cluster 2024-02-10T09:21:02.584283+0000 mon.a (mon.0) 480 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:21:03.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:03 smithi153 bash[39117]: cluster 2024-02-10T09:21:02.584313+0000 mon.a (mon.0) 481 : cluster [INF] Cluster is now healthy 2024-02-10T09:21:04.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:04 smithi204 bash[35809]: cluster 2024-02-10T09:21:03.782892+0000 mgr.y (mgr.14140) 142 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:04.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:04 smithi204 bash[35809]: audit 2024-02-10T09:21:04.064130+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:21:04.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:04 smithi204 bash[35809]: audit 2024-02-10T09:21:04.064870+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:21:04.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:04 smithi204 bash[35809]: audit 2024-02-10T09:21:04.065692+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.7", "key": "container_image"}]: dispatch 2024-02-10T09:21:04.938 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[35780]: cluster 2024-02-10T09:21:03.782892+0000 mgr.y (mgr.14140) 142 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[35780]: audit 2024-02-10T09:21:04.064130+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[35780]: audit 2024-02-10T09:21:04.064870+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[35780]: audit 2024-02-10T09:21:04.065692+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.7", "key": "container_image"}]: dispatch 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[39117]: cluster 2024-02-10T09:21:03.782892+0000 mgr.y (mgr.14140) 142 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[39117]: audit 2024-02-10T09:21:04.064130+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[39117]: audit 2024-02-10T09:21:04.064870+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:21:04.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:04 smithi153 bash[39117]: audit 2024-02-10T09:21:04.065692+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.7", "key": "container_image"}]: dispatch 2024-02-10T09:21:05.693 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:05 smithi204 bash[35809]: cephadm 2024-02-10T09:21:04.065418+0000 mgr.y (mgr.14140) 143 : cephadm [INF] Deploying daemon osd.7 on smithi204 2024-02-10T09:21:05.758 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 7 on host 'smithi204' 2024-02-10T09:21:05.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:05 smithi153 bash[35780]: cephadm 2024-02-10T09:21:04.065418+0000 mgr.y (mgr.14140) 143 : cephadm [INF] Deploying daemon osd.7 on smithi204 2024-02-10T09:21:05.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:05 smithi153 bash[39117]: cephadm 2024-02-10T09:21:04.065418+0000 mgr.y (mgr.14140) 143 : cephadm [INF] Deploying daemon osd.7 on smithi204 2024-02-10T09:21:06.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[35809]: audit 2024-02-10T09:21:05.751747+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:06.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[35809]: audit 2024-02-10T09:21:05.752213+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:21:06.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[35809]: audit 2024-02-10T09:21:05.754799+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:06.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[35809]: cluster 2024-02-10T09:21:05.783294+0000 mgr.y (mgr.14140) 144 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:06.675 DEBUG:teuthology.orchestra.run.smithi204:osd.7> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T09:21:06.678 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T09:21:06.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T09:21:06.709 INFO:journalctl@ceph.osd.7.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:21:06.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[35780]: audit 2024-02-10T09:21:05.751747+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:06.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[35780]: audit 2024-02-10T09:21:05.752213+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:21:06.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[39117]: audit 2024-02-10T09:21:05.751747+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:06.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[39117]: audit 2024-02-10T09:21:05.752213+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:21:06.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[35780]: audit 2024-02-10T09:21:05.754799+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:06.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[35780]: cluster 2024-02-10T09:21:05.783294+0000 mgr.y (mgr.14140) 144 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:06.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[39117]: audit 2024-02-10T09:21:05.754799+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162418\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162510\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162545\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162576\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:20:50.162605\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_device_update\": \"2024-02-10T09:20:55.264678\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:06.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:06 smithi153 bash[39117]: cluster 2024-02-10T09:21:05.783294+0000 mgr.y (mgr.14140) 144 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:07.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T09:21:07.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-02-10T09:21:07.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T09:21:07.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-02-10T09:21:07.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T09:21:07.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T09:21:07.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:06 smithi204 bash[42646]: --> ceph-volume lvm activate successful for osd ID: 7 2024-02-10T09:21:07.984 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:07 smithi153 bash[39117]: cluster 2024-02-10T09:21:07.783600+0000 mgr.y (mgr.14140) 145 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:07.984 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:07 smithi153 bash[35780]: cluster 2024-02-10T09:21:07.783600+0000 mgr.y (mgr.14140) 145 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:08.214 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:07 smithi204 bash[35809]: cluster 2024-02-10T09:21:07.783600+0000 mgr.y (mgr.14140) 145 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:21:08.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[39117]: audit 2024-02-10T09:21:08.601536+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[35780]: audit 2024-02-10T09:21:08.601536+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[35780]: audit 2024-02-10T09:21:08.604341+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[35780]: audit 2024-02-10T09:21:08.604873+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[35780]: audit 2024-02-10T09:21:08.623301+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.153:0/847783649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[39117]: audit 2024-02-10T09:21:08.604341+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[39117]: audit 2024-02-10T09:21:08.604873+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:21:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:08 smithi153 bash[39117]: audit 2024-02-10T09:21:08.623301+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.153:0/847783649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T09:21:09.220 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:08 smithi204 bash[42840]: debug 2024-02-10T09:21:08.900+0000 7fc103fefec0 -1 Falling back to public interface 2024-02-10T09:21:09.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:08 smithi204 bash[35809]: audit 2024-02-10T09:21:08.601536+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:09.221 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:08 smithi204 bash[35809]: audit 2024-02-10T09:21:08.604341+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"comment\": \"not used by ceph\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:09.221 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:08 smithi204 bash[35809]: audit 2024-02-10T09:21:08.604873+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2024-02-10T09:21:09.221 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:08 smithi204 bash[35809]: audit 2024-02-10T09:21:08.623301+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.153:0/847783649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T09:21:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":49,"num_osds":8,"num_up_osds":7,"osd_up_since":1707556853,"num_in_osds":7,"osd_in_since":1707556853,"num_remapped_pgs":0} 2024-02-10T09:21:09.576 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:09 smithi204 bash[42840]: debug 2024-02-10T09:21:09.450+0000 7fc103fefec0 -1 osd.7 0 log_to_monitors {default=true} 2024-02-10T09:21:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:09 smithi153 bash[35780]: audit 2024-02-10T09:21:09.454059+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:21:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:09 smithi153 bash[35780]: audit 2024-02-10T09:21:09.454388+0000 mon.a (mon.0) 491 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:21:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:09 smithi153 bash[35780]: cluster 2024-02-10T09:21:09.783854+0000 mgr.y (mgr.14140) 146 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:09 smithi153 bash[39117]: audit 2024-02-10T09:21:09.454059+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:21:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:09 smithi153 bash[39117]: audit 2024-02-10T09:21:09.454388+0000 mon.a (mon.0) 491 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:21:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:09 smithi153 bash[39117]: cluster 2024-02-10T09:21:09.783854+0000 mgr.y (mgr.14140) 146 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:10.197 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:09 smithi204 bash[35809]: audit 2024-02-10T09:21:09.454059+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:21:10.198 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:09 smithi204 bash[35809]: audit 2024-02-10T09:21:09.454388+0000 mon.a (mon.0) 491 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:21:10.198 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:09 smithi204 bash[35809]: cluster 2024-02-10T09:21:09.783854+0000 mgr.y (mgr.14140) 146 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:10.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T09:21:10.889 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: audit 2024-02-10T09:21:09.885007+0000 mon.a (mon.0) 492 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T09:21:10.889 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: cluster 2024-02-10T09:21:09.885078+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e50: 8 total, 7 up, 7 in 2024-02-10T09:21:10.889 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: audit 2024-02-10T09:21:09.885213+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:10.890 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: audit 2024-02-10T09:21:09.887006+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:21:10.890 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: audit 2024-02-10T09:21:09.887276+0000 mon.a (mon.0) 495 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:21:10.890 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: cluster 2024-02-10T09:21:10.437303+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:21:10.890 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: cluster 2024-02-10T09:21:10.437348+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:21:10.890 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: audit 2024-02-10T09:21:10.637989+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:21:11.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[39117]: audit 2024-02-10T09:21:10.641375+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: audit 2024-02-10T09:21:09.885007+0000 mon.a (mon.0) 492 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: cluster 2024-02-10T09:21:09.885078+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e50: 8 total, 7 up, 7 in 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: audit 2024-02-10T09:21:09.885213+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: audit 2024-02-10T09:21:09.887006+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: audit 2024-02-10T09:21:09.887276+0000 mon.a (mon.0) 495 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: cluster 2024-02-10T09:21:10.437303+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: cluster 2024-02-10T09:21:10.437348+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: audit 2024-02-10T09:21:10.637989+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:21:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:10 smithi153 bash[35780]: audit 2024-02-10T09:21:10.641375+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:21:11.331 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[42840]: debug 2024-02-10T09:21:10.889+0000 7fc0e8d9c700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T09:21:11.331 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[42840]: debug 2024-02-10T09:21:10.895+0000 7fc0f3fb8700 -1 osd.7 51 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-02-10T09:21:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: audit 2024-02-10T09:21:09.885007+0000 mon.a (mon.0) 492 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T09:21:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: cluster 2024-02-10T09:21:09.885078+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e50: 8 total, 7 up, 7 in 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: audit 2024-02-10T09:21:09.885213+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: audit 2024-02-10T09:21:09.887006+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: audit 2024-02-10T09:21:09.887276+0000 mon.a (mon.0) 495 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: cluster 2024-02-10T09:21:10.437303+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: cluster 2024-02-10T09:21:10.437348+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: audit 2024-02-10T09:21:10.637989+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:21:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:10 smithi204 bash[35809]: audit 2024-02-10T09:21:10.641375+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:21:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:21:12.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[35780]: audit 2024-02-10T09:21:10.885184+0000 mon.a (mon.0) 498 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[35780]: cluster 2024-02-10T09:21:10.885218+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e51: 8 total, 7 up, 7 in 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[35780]: audit 2024-02-10T09:21:10.885662+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[35780]: audit 2024-02-10T09:21:11.724416+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[35780]: cluster 2024-02-10T09:21:11.784088+0000 mgr.y (mgr.14140) 147 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[39117]: audit 2024-02-10T09:21:10.885184+0000 mon.a (mon.0) 498 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[39117]: cluster 2024-02-10T09:21:10.885218+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e51: 8 total, 7 up, 7 in 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[39117]: audit 2024-02-10T09:21:10.885662+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[39117]: audit 2024-02-10T09:21:11.724416+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:21:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:11 smithi153 bash[39117]: cluster 2024-02-10T09:21:11.784088+0000 mgr.y (mgr.14140) 147 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:11 smithi204 bash[35809]: audit 2024-02-10T09:21:10.885184+0000 mon.a (mon.0) 498 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-02-10T09:21:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:11 smithi204 bash[35809]: cluster 2024-02-10T09:21:10.885218+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e51: 8 total, 7 up, 7 in 2024-02-10T09:21:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:11 smithi204 bash[35809]: audit 2024-02-10T09:21:10.885662+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:11 smithi204 bash[35809]: audit 2024-02-10T09:21:11.724416+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:21:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:11 smithi204 bash[35809]: cluster 2024-02-10T09:21:11.784088+0000 mgr.y (mgr.14140) 147 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 3.1 MiB used, 619 GiB / 626 GiB avail 2024-02-10T09:21:12.720 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1707556871,"num_in_osds":8,"osd_in_since":1707556871,"num_remapped_pgs":0} 2024-02-10T09:21:12.723 INFO:tasks.cephadm:Adding prometheus.a on smithi204 2024-02-10T09:21:12.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi204=a' 2024-02-10T09:21:13.078 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:12 smithi204 bash[35809]: cluster 2024-02-10T09:21:11.890250+0000 mon.a (mon.0) 502 : cluster [INF] osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370] boot 2024-02-10T09:21:13.078 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:12 smithi204 bash[35809]: cluster 2024-02-10T09:21:11.890294+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-10T09:21:13.078 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:12 smithi204 bash[35809]: audit 2024-02-10T09:21:11.890929+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:13.078 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:12 smithi204 bash[35809]: audit 2024-02-10T09:21:11.912309+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:13.078 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:12 smithi204 bash[35809]: audit 2024-02-10T09:21:11.916004+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:13.078 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:12 smithi204 bash[35809]: audit 2024-02-10T09:21:12.102453+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.153:0/1759166606' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T09:21:13.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[35780]: cluster 2024-02-10T09:21:11.890250+0000 mon.a (mon.0) 502 : cluster [INF] osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370] boot 2024-02-10T09:21:13.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[35780]: cluster 2024-02-10T09:21:11.890294+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-10T09:21:13.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[35780]: audit 2024-02-10T09:21:11.890929+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:13.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[39117]: cluster 2024-02-10T09:21:11.890250+0000 mon.a (mon.0) 502 : cluster [INF] osd.7 [v2:172.21.15.204:6824/971867370,v1:172.21.15.204:6825/971867370] boot 2024-02-10T09:21:13.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[39117]: cluster 2024-02-10T09:21:11.890294+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-10T09:21:13.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[39117]: audit 2024-02-10T09:21:11.890929+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:21:13.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[35780]: audit 2024-02-10T09:21:11.912309+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:13.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[35780]: audit 2024-02-10T09:21:11.916004+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:13.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[35780]: audit 2024-02-10T09:21:12.102453+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.153:0/1759166606' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T09:21:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[39117]: audit 2024-02-10T09:21:11.912309+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:21:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[39117]: audit 2024-02-10T09:21:11.916004+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}}, \"last_daemon_update\": \"2024-02-10T09:21:08.600879\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:21:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:12 smithi153 bash[39117]: audit 2024-02-10T09:21:12.102453+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.153:0/1759166606' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T09:21:13.996 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:13 smithi204 bash[35809]: cluster 2024-02-10T09:21:12.899959+0000 mon.a (mon.0) 507 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-10T09:21:13.996 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:13 smithi204 bash[35809]: cluster 2024-02-10T09:21:13.784375+0000 mgr.y (mgr.14140) 148 : cluster [DBG] pgmap v158: 1 pgs: 1 peering; 0 B data, 3.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:13 smithi153 bash[35780]: cluster 2024-02-10T09:21:12.899959+0000 mon.a (mon.0) 507 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-10T09:21:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:13 smithi153 bash[35780]: cluster 2024-02-10T09:21:13.784375+0000 mgr.y (mgr.14140) 148 : cluster [DBG] pgmap v158: 1 pgs: 1 peering; 0 B data, 3.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:13 smithi153 bash[39117]: cluster 2024-02-10T09:21:12.899959+0000 mon.a (mon.0) 507 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-10T09:21:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:13 smithi153 bash[39117]: cluster 2024-02-10T09:21:13.784375+0000 mgr.y (mgr.14140) 148 : cluster [DBG] pgmap v158: 1 pgs: 1 peering; 0 B data, 3.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:14.690 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled prometheus update... 2024-02-10T09:21:15.037 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:14 smithi204 bash[35809]: cluster 2024-02-10T09:21:13.898255+0000 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:21:15.038 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:14 smithi204 bash[35809]: audit 2024-02-10T09:21:14.684793+0000 mgr.y (mgr.14140) 149 : audit [DBG] from='client.34319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:15.038 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:14 smithi204 bash[35809]: cephadm 2024-02-10T09:21:14.685395+0000 mgr.y (mgr.14140) 150 : cephadm [INF] Saving service prometheus spec with placement count:1 smithi204=a 2024-02-10T09:21:15.038 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:14 smithi204 bash[35809]: audit 2024-02-10T09:21:14.685756+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.prometheus","val":"{\"created\": \"2024-02-10T09:21:14.685407\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"prometheus\"}}"}]: dispatch 2024-02-10T09:21:15.038 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:14 smithi204 bash[35809]: audit 2024-02-10T09:21:14.688741+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.prometheus","val":"{\"created\": \"2024-02-10T09:21:14.685407\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"prometheus\"}}"}]': finished 2024-02-10T09:21:15.038 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:14 smithi204 bash[35809]: cephadm 2024-02-10T09:21:14.692923+0000 mgr.y (mgr.14140) 151 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:21:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[35780]: cluster 2024-02-10T09:21:13.898255+0000 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:21:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[35780]: audit 2024-02-10T09:21:14.684793+0000 mgr.y (mgr.14140) 149 : audit [DBG] from='client.34319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[35780]: cephadm 2024-02-10T09:21:14.685395+0000 mgr.y (mgr.14140) 150 : cephadm [INF] Saving service prometheus spec with placement count:1 smithi204=a 2024-02-10T09:21:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[35780]: audit 2024-02-10T09:21:14.685756+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.prometheus","val":"{\"created\": \"2024-02-10T09:21:14.685407\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"prometheus\"}}"}]: dispatch 2024-02-10T09:21:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[35780]: audit 2024-02-10T09:21:14.688741+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.prometheus","val":"{\"created\": \"2024-02-10T09:21:14.685407\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"prometheus\"}}"}]': finished 2024-02-10T09:21:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[35780]: cephadm 2024-02-10T09:21:14.692923+0000 mgr.y (mgr.14140) 151 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:21:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[39117]: cluster 2024-02-10T09:21:13.898255+0000 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:21:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[39117]: audit 2024-02-10T09:21:14.684793+0000 mgr.y (mgr.14140) 149 : audit [DBG] from='client.34319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[39117]: cephadm 2024-02-10T09:21:14.685395+0000 mgr.y (mgr.14140) 150 : cephadm [INF] Saving service prometheus spec with placement count:1 smithi204=a 2024-02-10T09:21:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[39117]: audit 2024-02-10T09:21:14.685756+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.prometheus","val":"{\"created\": \"2024-02-10T09:21:14.685407\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"prometheus\"}}"}]: dispatch 2024-02-10T09:21:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[39117]: audit 2024-02-10T09:21:14.688741+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.prometheus","val":"{\"created\": \"2024-02-10T09:21:14.685407\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"prometheus\"}}"}]': finished 2024-02-10T09:21:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:14 smithi153 bash[39117]: cephadm 2024-02-10T09:21:14.692923+0000 mgr.y (mgr.14140) 151 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:21:15.380 DEBUG:teuthology.orchestra.run.smithi204:prometheus.a> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T09:21:15.382 INFO:tasks.cephadm:Adding node-exporter.a on smithi153 2024-02-10T09:21:15.382 INFO:tasks.cephadm:Adding node-exporter.b on smithi204 2024-02-10T09:21:15.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi153=a;smithi204=b' 2024-02-10T09:21:15.410 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:21:16.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:15 smithi204 bash[35809]: cluster 2024-02-10T09:21:15.784703+0000 mgr.y (mgr.14140) 152 : cluster [DBG] pgmap v159: 1 pgs: 1 peering; 0 B data, 3.8 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:15 smithi153 bash[39117]: cluster 2024-02-10T09:21:15.784703+0000 mgr.y (mgr.14140) 152 : cluster [DBG] pgmap v159: 1 pgs: 1 peering; 0 B data, 3.8 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:15 smithi153 bash[35780]: cluster 2024-02-10T09:21:15.784703+0000 mgr.y (mgr.14140) 152 : cluster [DBG] pgmap v159: 1 pgs: 1 peering; 0 B data, 3.8 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:17.193 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled node-exporter update... 2024-02-10T09:21:17.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:17 smithi204 bash[35809]: audit 2024-02-10T09:21:17.188973+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.node-exporter","val":"{\"created\": \"2024-02-10T09:21:17.188578\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"b\", \"network\": \"\"}]}, \"service_type\": \"node-exporter\"}}"}]: dispatch 2024-02-10T09:21:17.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:17 smithi204 bash[35809]: audit 2024-02-10T09:21:17.192088+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.node-exporter","val":"{\"created\": \"2024-02-10T09:21:17.188578\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"b\", \"network\": \"\"}]}, \"service_type\": \"node-exporter\"}}"}]': finished 2024-02-10T09:21:17.892 DEBUG:teuthology.orchestra.run.smithi153:node-exporter.a> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T09:21:17.895 DEBUG:teuthology.orchestra.run.smithi204:node-exporter.b> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T09:21:17.897 INFO:tasks.cephadm:Adding alertmanager.a on smithi153 2024-02-10T09:21:17.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi153=a' 2024-02-10T09:21:17.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:17 smithi153 bash[39117]: audit 2024-02-10T09:21:17.188973+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.node-exporter","val":"{\"created\": \"2024-02-10T09:21:17.188578\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"b\", \"network\": \"\"}]}, \"service_type\": \"node-exporter\"}}"}]: dispatch 2024-02-10T09:21:17.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:17 smithi153 bash[39117]: audit 2024-02-10T09:21:17.192088+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.node-exporter","val":"{\"created\": \"2024-02-10T09:21:17.188578\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"b\", \"network\": \"\"}]}, \"service_type\": \"node-exporter\"}}"}]': finished 2024-02-10T09:21:17.926 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:17 smithi153 bash[35780]: audit 2024-02-10T09:21:17.188973+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.node-exporter","val":"{\"created\": \"2024-02-10T09:21:17.188578\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"b\", \"network\": \"\"}]}, \"service_type\": \"node-exporter\"}}"}]: dispatch 2024-02-10T09:21:17.926 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:17 smithi153 bash[35780]: audit 2024-02-10T09:21:17.192088+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.node-exporter","val":"{\"created\": \"2024-02-10T09:21:17.188578\", \"spec\": {\"placement\": {\"count\": 2, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}, {\"hostname\": \"smithi204\", \"name\": \"b\", \"network\": \"\"}]}, \"service_type\": \"node-exporter\"}}"}]': finished 2024-02-10T09:21:17.927 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:21:17.932 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:21:18.865 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:18 smithi204 bash[35809]: audit 2024-02-10T09:21:17.187940+0000 mgr.y (mgr.14140) 153 : audit [DBG] from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi153=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:18.865 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:18 smithi204 bash[35809]: cephadm 2024-02-10T09:21:17.188562+0000 mgr.y (mgr.14140) 154 : cephadm [INF] Saving service node-exporter spec with placement count:2 smithi153=a,smithi204=b 2024-02-10T09:21:18.865 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:18 smithi204 bash[35809]: cluster 2024-02-10T09:21:17.785051+0000 mgr.y (mgr.14140) 155 : cluster [DBG] pgmap v160: 1 pgs: 1 peering; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:18 smithi153 bash[39117]: audit 2024-02-10T09:21:17.187940+0000 mgr.y (mgr.14140) 153 : audit [DBG] from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi153=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:18 smithi153 bash[39117]: cephadm 2024-02-10T09:21:17.188562+0000 mgr.y (mgr.14140) 154 : cephadm [INF] Saving service node-exporter spec with placement count:2 smithi153=a,smithi204=b 2024-02-10T09:21:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:18 smithi153 bash[39117]: cluster 2024-02-10T09:21:17.785051+0000 mgr.y (mgr.14140) 155 : cluster [DBG] pgmap v160: 1 pgs: 1 peering; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:18 smithi153 bash[35780]: audit 2024-02-10T09:21:17.187940+0000 mgr.y (mgr.14140) 153 : audit [DBG] from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi153=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:18 smithi153 bash[35780]: cephadm 2024-02-10T09:21:17.188562+0000 mgr.y (mgr.14140) 154 : cephadm [INF] Saving service node-exporter spec with placement count:2 smithi153=a,smithi204=b 2024-02-10T09:21:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:18 smithi153 bash[35780]: cluster 2024-02-10T09:21:17.785051+0000 mgr.y (mgr.14140) 155 : cluster [DBG] pgmap v160: 1 pgs: 1 peering; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:19.796 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled alertmanager update... 2024-02-10T09:21:20.180 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: cluster 2024-02-10T09:21:19.785391+0000 mgr.y (mgr.14140) 156 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:20.180 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: audit 2024-02-10T09:21:19.790332+0000 mgr.y (mgr.14140) 157 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:20.180 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: cephadm 2024-02-10T09:21:19.790916+0000 mgr.y (mgr.14140) 158 : cephadm [INF] Saving service alertmanager spec with placement count:1 smithi153=a 2024-02-10T09:21:20.180 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: audit 2024-02-10T09:21:19.791353+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.alertmanager","val":"{\"created\": \"2024-02-10T09:21:19.790933\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"alertmanager\"}}"}]: dispatch 2024-02-10T09:21:20.180 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: audit 2024-02-10T09:21:19.794543+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.alertmanager","val":"{\"created\": \"2024-02-10T09:21:19.790933\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"alertmanager\"}}"}]': finished 2024-02-10T09:21:20.181 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: cluster 2024-02-10T09:21:19.836027+0000 mon.a (mon.0) 515 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:21:20.181 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:19 smithi204 bash[35809]: cluster 2024-02-10T09:21:19.836056+0000 mon.a (mon.0) 516 : cluster [INF] Cluster is now healthy 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: cluster 2024-02-10T09:21:19.785391+0000 mgr.y (mgr.14140) 156 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: audit 2024-02-10T09:21:19.790332+0000 mgr.y (mgr.14140) 157 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: cephadm 2024-02-10T09:21:19.790916+0000 mgr.y (mgr.14140) 158 : cephadm [INF] Saving service alertmanager spec with placement count:1 smithi153=a 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: audit 2024-02-10T09:21:19.791353+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.alertmanager","val":"{\"created\": \"2024-02-10T09:21:19.790933\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"alertmanager\"}}"}]: dispatch 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: audit 2024-02-10T09:21:19.794543+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.alertmanager","val":"{\"created\": \"2024-02-10T09:21:19.790933\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"alertmanager\"}}"}]': finished 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: cluster 2024-02-10T09:21:19.836027+0000 mon.a (mon.0) 515 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[39117]: cluster 2024-02-10T09:21:19.836056+0000 mon.a (mon.0) 516 : cluster [INF] Cluster is now healthy 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: cluster 2024-02-10T09:21:19.785391+0000 mgr.y (mgr.14140) 156 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: audit 2024-02-10T09:21:19.790332+0000 mgr.y (mgr.14140) 157 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: cephadm 2024-02-10T09:21:19.790916+0000 mgr.y (mgr.14140) 158 : cephadm [INF] Saving service alertmanager spec with placement count:1 smithi153=a 2024-02-10T09:21:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: audit 2024-02-10T09:21:19.791353+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.alertmanager","val":"{\"created\": \"2024-02-10T09:21:19.790933\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"alertmanager\"}}"}]: dispatch 2024-02-10T09:21:20.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: audit 2024-02-10T09:21:19.794543+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.alertmanager","val":"{\"created\": \"2024-02-10T09:21:19.790933\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi153\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"alertmanager\"}}"}]': finished 2024-02-10T09:21:20.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: cluster 2024-02-10T09:21:19.836027+0000 mon.a (mon.0) 515 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:21:20.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:19 smithi153 bash[35780]: cluster 2024-02-10T09:21:19.836056+0000 mon.a (mon.0) 516 : cluster [INF] Cluster is now healthy 2024-02-10T09:21:20.507 DEBUG:teuthology.orchestra.run.smithi153:alertmanager.a> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T09:21:20.510 INFO:tasks.cephadm:Adding grafana.a on smithi204 2024-02-10T09:21:20.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi204=a' 2024-02-10T09:21:20.543 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:-- Logs begin at Sat 2024-02-10 09:06:34 UTC. -- 2024-02-10T09:21:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:20 smithi153 bash[35780]: audit 2024-02-10T09:21:20.642471+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556871.8922586, \"finished_at\": 1707556879.842647}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:21:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:20 smithi153 bash[35780]: audit 2024-02-10T09:21:20.645774+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556871.8922586, \"finished_at\": 1707556879.842647}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:21:21.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:20 smithi153 bash[39117]: audit 2024-02-10T09:21:20.642471+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556871.8922586, \"finished_at\": 1707556879.842647}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:21:21.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:20 smithi153 bash[39117]: audit 2024-02-10T09:21:20.645774+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556871.8922586, \"finished_at\": 1707556879.842647}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:21:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:20 smithi204 bash[35809]: audit 2024-02-10T09:21:20.642471+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556871.8922586, \"finished_at\": 1707556879.842647}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:21:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:20 smithi204 bash[35809]: audit 2024-02-10T09:21:20.645774+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556815.599134, \"finished_at\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556834.5899186, \"finished_at\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556853.9054804, \"finished_at\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556871.8922586, \"finished_at\": 1707556879.842647}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:21:22.442 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled grafana update... 2024-02-10T09:21:22.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:22 smithi204 bash[35809]: cluster 2024-02-10T09:21:21.785697+0000 mgr.y (mgr.14140) 159 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:22.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:22 smithi204 bash[35809]: audit 2024-02-10T09:21:22.438096+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.grafana","val":"{\"created\": \"2024-02-10T09:21:22.437749\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"grafana\"}}"}]: dispatch 2024-02-10T09:21:22.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:22 smithi204 bash[35809]: audit 2024-02-10T09:21:22.441457+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.grafana","val":"{\"created\": \"2024-02-10T09:21:22.437749\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"grafana\"}}"}]': finished 2024-02-10T09:21:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:22 smithi153 bash[35780]: cluster 2024-02-10T09:21:21.785697+0000 mgr.y (mgr.14140) 159 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:22 smithi153 bash[35780]: audit 2024-02-10T09:21:22.438096+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.grafana","val":"{\"created\": \"2024-02-10T09:21:22.437749\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"grafana\"}}"}]: dispatch 2024-02-10T09:21:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:22 smithi153 bash[35780]: audit 2024-02-10T09:21:22.441457+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.grafana","val":"{\"created\": \"2024-02-10T09:21:22.437749\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"grafana\"}}"}]': finished 2024-02-10T09:21:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:22 smithi153 bash[39117]: cluster 2024-02-10T09:21:21.785697+0000 mgr.y (mgr.14140) 159 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:22 smithi153 bash[39117]: audit 2024-02-10T09:21:22.438096+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.grafana","val":"{\"created\": \"2024-02-10T09:21:22.437749\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"grafana\"}}"}]: dispatch 2024-02-10T09:21:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:22 smithi153 bash[39117]: audit 2024-02-10T09:21:22.441457+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.grafana","val":"{\"created\": \"2024-02-10T09:21:22.437749\", \"spec\": {\"placement\": {\"count\": 1, \"hosts\": [{\"hostname\": \"smithi204\", \"name\": \"a\", \"network\": \"\"}]}, \"service_type\": \"grafana\"}}"}]': finished 2024-02-10T09:21:23.084 DEBUG:teuthology.orchestra.run.smithi204:grafana.a> sudo journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T09:21:23.086 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T09:21:23.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T09:21:23.116 INFO:journalctl@ceph.grafana.a.smithi204.stdout:-- Logs begin at Sat 2024-02-10 09:06:37 UTC. -- 2024-02-10T09:21:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:23 smithi153 bash[35780]: audit 2024-02-10T09:21:22.437075+0000 mgr.y (mgr.14140) 160 : audit [DBG] from='client.34337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:23 smithi153 bash[35780]: cephadm 2024-02-10T09:21:22.437723+0000 mgr.y (mgr.14140) 161 : cephadm [INF] Saving service grafana spec with placement count:1 smithi204=a 2024-02-10T09:21:23.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:23 smithi153 bash[39117]: audit 2024-02-10T09:21:22.437075+0000 mgr.y (mgr.14140) 160 : audit [DBG] from='client.34337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:23.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:23 smithi153 bash[39117]: cephadm 2024-02-10T09:21:22.437723+0000 mgr.y (mgr.14140) 161 : cephadm [INF] Saving service grafana spec with placement count:1 smithi204=a 2024-02-10T09:21:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:23 smithi204 bash[35809]: audit 2024-02-10T09:21:22.437075+0000 mgr.y (mgr.14140) 160 : audit [DBG] from='client.34337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:23 smithi204 bash[35809]: cephadm 2024-02-10T09:21:22.437723+0000 mgr.y (mgr.14140) 161 : cephadm [INF] Saving service grafana spec with placement count:1 smithi204=a 2024-02-10T09:21:24.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:24 smithi153 bash[35780]: cluster 2024-02-10T09:21:23.785999+0000 mgr.y (mgr.14140) 162 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:24.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:24 smithi153 bash[39117]: cluster 2024-02-10T09:21:23.785999+0000 mgr.y (mgr.14140) 162 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:25.023 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-02-10T09:21:25.023 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAVQMdlgkAwARAAVsffmbkcWwOJPuC/EkEpFQ== 2024-02-10T09:21:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:24 smithi204 bash[35809]: cluster 2024-02-10T09:21:23.785999+0000 mgr.y (mgr.14140) 162 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:25.688 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:25 smithi153 bash[39117]: audit 2024-02-10T09:21:25.019807+0000 mon.a (mon.0) 521 : audit [INF] from='client.? 172.21.15.153:0/635567111' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:25.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:25 smithi153 bash[39117]: audit 2024-02-10T09:21:25.022890+0000 mon.a (mon.0) 522 : audit [INF] from='client.? 172.21.15.153:0/635567111' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T09:21:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:25 smithi153 bash[35780]: audit 2024-02-10T09:21:25.019807+0000 mon.a (mon.0) 521 : audit [INF] from='client.? 172.21.15.153:0/635567111' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:25 smithi153 bash[35780]: audit 2024-02-10T09:21:25.022890+0000 mon.a (mon.0) 522 : audit [INF] from='client.? 172.21.15.153:0/635567111' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T09:21:25.707 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-02-10T09:21:25.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T09:21:25.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T09:21:25.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T09:21:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:25 smithi204 bash[35809]: audit 2024-02-10T09:21:25.019807+0000 mon.a (mon.0) 521 : audit [INF] from='client.? 172.21.15.153:0/635567111' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:25 smithi204 bash[35809]: audit 2024-02-10T09:21:25.022890+0000 mon.a (mon.0) 522 : audit [INF] from='client.? 172.21.15.153:0/635567111' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T09:21:26.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:26 smithi204 bash[35809]: cluster 2024-02-10T09:21:25.786251+0000 mgr.y (mgr.14140) 163 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:26 smithi153 bash[39117]: cluster 2024-02-10T09:21:25.786251+0000 mgr.y (mgr.14140) 163 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:26.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:26 smithi153 bash[35780]: cluster 2024-02-10T09:21:25.786251+0000 mgr.y (mgr.14140) 163 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:27.810 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2024-02-10T09:21:27.810 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAXQMdlIHkJMBAAliLPTtDL4Wyf/Fa+zHbA6A== 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[35780]: cluster 2024-02-10T09:21:27.786572+0000 mgr.y (mgr.14140) 164 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[35780]: audit 2024-02-10T09:21:27.805340+0000 mon.b (mon.2) 20 : audit [INF] from='client.? 172.21.15.204:0/1597668264' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[35780]: audit 2024-02-10T09:21:27.805781+0000 mon.a (mon.0) 523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[35780]: audit 2024-02-10T09:21:27.809295+0000 mon.a (mon.0) 524 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[39117]: cluster 2024-02-10T09:21:27.786572+0000 mgr.y (mgr.14140) 164 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[39117]: audit 2024-02-10T09:21:27.805340+0000 mon.b (mon.2) 20 : audit [INF] from='client.? 172.21.15.204:0/1597668264' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[39117]: audit 2024-02-10T09:21:27.805781+0000 mon.a (mon.0) 523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:27 smithi153 bash[39117]: audit 2024-02-10T09:21:27.809295+0000 mon.a (mon.0) 524 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T09:21:28.214 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:27 smithi204 bash[35809]: cluster 2024-02-10T09:21:27.786572+0000 mgr.y (mgr.14140) 164 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:28.215 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:27 smithi204 bash[35809]: audit 2024-02-10T09:21:27.805340+0000 mon.b (mon.2) 20 : audit [INF] from='client.? 172.21.15.204:0/1597668264' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:28.215 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:27 smithi204 bash[35809]: audit 2024-02-10T09:21:27.805781+0000 mon.a (mon.0) 523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T09:21:28.215 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:27 smithi204 bash[35809]: audit 2024-02-10T09:21:27.809295+0000 mon.a (mon.0) 524 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T09:21:28.497 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-02-10T09:21:28.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T09:21:28.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T09:21:28.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T09:21:30.023 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:29 smithi153 bash[35780]: cluster 2024-02-10T09:21:29.786843+0000 mgr.y (mgr.14140) 165 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:30.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:29 smithi153 bash[39117]: cluster 2024-02-10T09:21:29.786843+0000 mgr.y (mgr.14140) 165 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:29 smithi204 bash[35809]: cluster 2024-02-10T09:21:29.786843+0000 mgr.y (mgr.14140) 165 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:30.373 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:21:30.373 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":53,"fsid":"09b43244-c7f5-11ee-95b8-87774f69a715","created":"2024-02-10T09:16:42.107493+0000","modified":"2024-02-10T09:21:12.893128+0000","last_up_change":"2024-02-10T09:21:11.886623+0000","last_in_change":"2024-02-10T09:21:11.886623+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T09:16:51.902036+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8351789c-a971-44a1-aa2f-b16e87fb72a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":12,"up_from":14,"up_thru":51,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6803","nonce":2774397228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6805","nonce":2774397228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6809","nonce":2774397228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6807","nonce":2774397228}]},"public_addr":"172.21.15.153:6803/2774397228","cluster_addr":"172.21.15.153:6805/2774397228","heartbeat_back_addr":"172.21.15.153:6809/2774397228","heartbeat_front_addr":"172.21.15.153:6807/2774397228","state":["exists","up"]},{"osd":1,"uuid":"fa508d13-ae9b-4d62-b567-1be1a9d46d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":20,"up_from":23,"up_thru":37,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6811","nonce":1003292780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6813","nonce":1003292780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6817","nonce":1003292780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6815","nonce":1003292780}]},"public_addr":"172.21.15.153:6811/1003292780","cluster_addr":"172.21.15.153:6813/1003292780","heartbeat_back_addr":"172.21.15.153:6817/1003292780","heartbeat_front_addr":"172.21.15.153:6815/1003292780","state":["exists","up"]},{"osd":2,"uuid":"b6e5a960-d7e8-4e2c-b913-6848f7f0de36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6819","nonce":2495526904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6821","nonce":2495526904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6825","nonce":2495526904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6823","nonce":2495526904}]},"public_addr":"172.21.15.153:6819/2495526904","cluster_addr":"172.21.15.153:6821/2495526904","heartbeat_back_addr":"172.21.15.153:6825/2495526904","heartbeat_front_addr":"172.21.15.153:6823/2495526904","state":["exists","up"]},{"osd":3,"uuid":"679b994c-2b1f-4f4d-9b26-64a3d365a094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6827","nonce":918480920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6829","nonce":918480920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6833","nonce":918480920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6831","nonce":918480920}]},"public_addr":"172.21.15.153:6827/918480920","cluster_addr":"172.21.15.153:6829/918480920","heartbeat_back_addr":"172.21.15.153:6833/918480920","heartbeat_front_addr":"172.21.15.153:6831/918480920","state":["exists","up"]},{"osd":4,"uuid":"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6801","nonce":1147358970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6803","nonce":1147358970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6807","nonce":1147358970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6805","nonce":1147358970}]},"public_addr":"172.21.15.204:6801/1147358970","cluster_addr":"172.21.15.204:6803/1147358970","heartbeat_back_addr":"172.21.15.204:6807/1147358970","heartbeat_front_addr":"172.21.15.204:6805/1147358970","state":["exists","up"]},{"osd":5,"uuid":"3ed1e885-ba58-4d66-a134-e63d325d4fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6809","nonce":4030840539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6811","nonce":4030840539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6815","nonce":4030840539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6813","nonce":4030840539}]},"public_addr":"172.21.15.204:6809/4030840539","cluster_addr":"172.21.15.204:6811/4030840539","heartbeat_back_addr":"172.21.15.204:6815/4030840539","heartbeat_front_addr":"172.21.15.204:6813/4030840539","state":["exists","up"]},{"osd":6,"uuid":"2f3e1b43-d7dd-44b2-be39-d240e518a51c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6817","nonce":2623910292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6819","nonce":2623910292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6823","nonce":2623910292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6821","nonce":2623910292}]},"public_addr":"172.21.15.204:6817/2623910292","cluster_addr":"172.21.15.204:6819/2623910292","heartbeat_back_addr":"172.21.15.204:6823/2623910292","heartbeat_front_addr":"172.21.15.204:6821/2623910292","state":["exists","up"]},{"osd":7,"uuid":"d38f33ce-8c0b-4235-be4f-4169e07e4cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6825","nonce":971867370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6827","nonce":971867370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6831","nonce":971867370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6829","nonce":971867370}]},"public_addr":"172.21.15.204:6825/971867370","cluster_addr":"172.21.15.204:6827/971867370","heartbeat_back_addr":"172.21.15.204:6831/971867370","heartbeat_front_addr":"172.21.15.204:6829/971867370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-02-10T09:19:16.704450+0000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:18:58.903499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:19:22.123354+0000","dead_epoch":21},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:19:41.394864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:00.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:14.414485+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:33.299242+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:52.131341+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:21:10.437350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.153:0/2708898466":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/1486748846":"2024-02-11T09:17:20.528659+0000","172.21.15.153:6801/1546421638":"2024-02-11T09:17:20.528659+0000","172.21.15.153:6800/1546421638":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/94314011":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/1500108103":"2024-02-11T09:16:59.319422+0000","172.21.15.153:0/782778237":"2024-02-11T09:16:59.319422+0000","172.21.15.153:6800/1824985332":"2024-02-11T09:16:59.319422+0000","172.21.15.153:0/1431131370":"2024-02-11T09:16:59.319422+0000","172.21.15.153:6801/1824985332":"2024-02-11T09:16:59.319422+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2024-02-10T09:21:31.067 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:30 smithi153 bash[35780]: audit 2024-02-10T09:21:30.371677+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.21.15.153:0/765312425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:31.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:30 smithi153 bash[39117]: audit 2024-02-10T09:21:30.371677+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.21.15.153:0/765312425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:31.078 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T09:16:51.902036+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-02-10T09:21:31.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-10T09:21:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:30 smithi204 bash[35809]: audit 2024-02-10T09:21:30.371677+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.21.15.153:0/765312425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:32.675 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:32 smithi153 bash[35780]: cluster 2024-02-10T09:21:31.787143+0000 mgr.y (mgr.14140) 166 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:32.675 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:32 smithi153 bash[39117]: cluster 2024-02-10T09:21:31.787143+0000 mgr.y (mgr.14140) 166 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:32 smithi204 bash[35809]: cluster 2024-02-10T09:21:31.787143+0000 mgr.y (mgr.14140) 166 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:33.281 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-02-10T09:21:33.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:33 smithi153 bash[35780]: audit 2024-02-10T09:21:33.280662+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.153:0/3296578706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-10T09:21:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:33 smithi153 bash[39117]: audit 2024-02-10T09:21:33.280662+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.153:0/3296578706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-10T09:21:33.974 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T09:21:33.974 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T09:21:33.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T09:21:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:33 smithi204 bash[35809]: audit 2024-02-10T09:21:33.280662+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.153:0/3296578706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-10T09:21:34.604 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:34 smithi153 bash[35780]: cluster 2024-02-10T09:21:33.787421+0000 mgr.y (mgr.14140) 167 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:34.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:34 smithi153 bash[39117]: cluster 2024-02-10T09:21:33.787421+0000 mgr.y (mgr.14140) 167 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:35.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:34 smithi204 bash[35809]: cluster 2024-02-10T09:21:33.787421+0000 mgr.y (mgr.14140) 167 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:36.058 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:21:36.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:35 smithi153 bash[35780]: cluster 2024-02-10T09:21:35.787677+0000 mgr.y (mgr.14140) 168 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:36.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:35 smithi153 bash[39117]: cluster 2024-02-10T09:21:35.787677+0000 mgr.y (mgr.14140) 168 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:35 smithi204 bash[35809]: cluster 2024-02-10T09:21:35.787677+0000 mgr.y (mgr.14140) 168 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:36.812 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":14,"active_gid":14140,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":677103091},{"type":"v1","addr":"172.21.15.153:6801","nonce":677103091}]},"active_addr":"172.21.15.153:6801/677103091","active_change":"2024-02-10T09:17:20.528980+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":34109,"name":"x","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi153.front.sepia.ceph.com:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1224574477}]},{"addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2595700759}]},{"addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3188946608}]}]}} 2024-02-10T09:21:36.815 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T09:21:36.815 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T09:21:36.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T09:21:37.141 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:36 smithi153 bash[39117]: audit 2024-02-10T09:21:36.054142+0000 mon.a (mon.0) 526 : audit [DBG] from='client.? 172.21.15.153:0/885483783' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:21:37.141 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:36 smithi153 bash[35780]: audit 2024-02-10T09:21:36.054142+0000 mon.a (mon.0) 526 : audit [DBG] from='client.? 172.21.15.153:0/885483783' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:21:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:36 smithi204 bash[35809]: audit 2024-02-10T09:21:36.054142+0000 mon.a (mon.0) 526 : audit [DBG] from='client.? 172.21.15.153:0/885483783' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:21:38.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:37 smithi153 bash[35780]: cluster 2024-02-10T09:21:37.787864+0000 mgr.y (mgr.14140) 169 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:38.176 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:37 smithi153 bash[39117]: cluster 2024-02-10T09:21:37.787864+0000 mgr.y (mgr.14140) 169 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:37 smithi204 bash[35809]: cluster 2024-02-10T09:21:37.787864+0000 mgr.y (mgr.14140) 169 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:38.756 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:21:38.756 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":53,"fsid":"09b43244-c7f5-11ee-95b8-87774f69a715","created":"2024-02-10T09:16:42.107493+0000","modified":"2024-02-10T09:21:12.893128+0000","last_up_change":"2024-02-10T09:21:11.886623+0000","last_in_change":"2024-02-10T09:21:11.886623+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T09:16:51.902036+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8351789c-a971-44a1-aa2f-b16e87fb72a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":12,"up_from":14,"up_thru":51,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6803","nonce":2774397228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6805","nonce":2774397228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6809","nonce":2774397228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6807","nonce":2774397228}]},"public_addr":"172.21.15.153:6803/2774397228","cluster_addr":"172.21.15.153:6805/2774397228","heartbeat_back_addr":"172.21.15.153:6809/2774397228","heartbeat_front_addr":"172.21.15.153:6807/2774397228","state":["exists","up"]},{"osd":1,"uuid":"fa508d13-ae9b-4d62-b567-1be1a9d46d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":20,"up_from":23,"up_thru":37,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6811","nonce":1003292780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6813","nonce":1003292780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6817","nonce":1003292780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6815","nonce":1003292780}]},"public_addr":"172.21.15.153:6811/1003292780","cluster_addr":"172.21.15.153:6813/1003292780","heartbeat_back_addr":"172.21.15.153:6817/1003292780","heartbeat_front_addr":"172.21.15.153:6815/1003292780","state":["exists","up"]},{"osd":2,"uuid":"b6e5a960-d7e8-4e2c-b913-6848f7f0de36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6819","nonce":2495526904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6821","nonce":2495526904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6825","nonce":2495526904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6823","nonce":2495526904}]},"public_addr":"172.21.15.153:6819/2495526904","cluster_addr":"172.21.15.153:6821/2495526904","heartbeat_back_addr":"172.21.15.153:6825/2495526904","heartbeat_front_addr":"172.21.15.153:6823/2495526904","state":["exists","up"]},{"osd":3,"uuid":"679b994c-2b1f-4f4d-9b26-64a3d365a094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6827","nonce":918480920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6829","nonce":918480920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6833","nonce":918480920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6831","nonce":918480920}]},"public_addr":"172.21.15.153:6827/918480920","cluster_addr":"172.21.15.153:6829/918480920","heartbeat_back_addr":"172.21.15.153:6833/918480920","heartbeat_front_addr":"172.21.15.153:6831/918480920","state":["exists","up"]},{"osd":4,"uuid":"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6801","nonce":1147358970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6803","nonce":1147358970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6807","nonce":1147358970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6805","nonce":1147358970}]},"public_addr":"172.21.15.204:6801/1147358970","cluster_addr":"172.21.15.204:6803/1147358970","heartbeat_back_addr":"172.21.15.204:6807/1147358970","heartbeat_front_addr":"172.21.15.204:6805/1147358970","state":["exists","up"]},{"osd":5,"uuid":"3ed1e885-ba58-4d66-a134-e63d325d4fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6809","nonce":4030840539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6811","nonce":4030840539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6815","nonce":4030840539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6813","nonce":4030840539}]},"public_addr":"172.21.15.204:6809/4030840539","cluster_addr":"172.21.15.204:6811/4030840539","heartbeat_back_addr":"172.21.15.204:6815/4030840539","heartbeat_front_addr":"172.21.15.204:6813/4030840539","state":["exists","up"]},{"osd":6,"uuid":"2f3e1b43-d7dd-44b2-be39-d240e518a51c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6817","nonce":2623910292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6819","nonce":2623910292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6823","nonce":2623910292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6821","nonce":2623910292}]},"public_addr":"172.21.15.204:6817/2623910292","cluster_addr":"172.21.15.204:6819/2623910292","heartbeat_back_addr":"172.21.15.204:6823/2623910292","heartbeat_front_addr":"172.21.15.204:6821/2623910292","state":["exists","up"]},{"osd":7,"uuid":"d38f33ce-8c0b-4235-be4f-4169e07e4cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6825","nonce":971867370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6827","nonce":971867370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6831","nonce":971867370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6829","nonce":971867370}]},"public_addr":"172.21.15.204:6825/971867370","cluster_addr":"172.21.15.204:6827/971867370","heartbeat_back_addr":"172.21.15.204:6831/971867370","heartbeat_front_addr":"172.21.15.204:6829/971867370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-02-10T09:19:16.704450+0000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:18:58.903499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:19:22.123354+0000","dead_epoch":21},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:19:41.394864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:00.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:14.414485+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:33.299242+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:52.131341+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:21:10.437350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.153:0/2708898466":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/1486748846":"2024-02-11T09:17:20.528659+0000","172.21.15.153:6801/1546421638":"2024-02-11T09:17:20.528659+0000","172.21.15.153:6800/1546421638":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/94314011":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/1500108103":"2024-02-11T09:16:59.319422+0000","172.21.15.153:0/782778237":"2024-02-11T09:16:59.319422+0000","172.21.15.153:6800/1824985332":"2024-02-11T09:16:59.319422+0000","172.21.15.153:0/1431131370":"2024-02-11T09:16:59.319422+0000","172.21.15.153:6801/1824985332":"2024-02-11T09:16:59.319422+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2024-02-10T09:21:39.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:38 smithi153 bash[39117]: audit 2024-02-10T09:21:38.755421+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.153:0/555254160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:39.101 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:38 smithi153 bash[35780]: audit 2024-02-10T09:21:38.755421+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.153:0/555254160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:39.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:38 smithi204 bash[35809]: audit 2024-02-10T09:21:38.755421+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.153:0/555254160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:39.510 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T09:21:39.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T09:21:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:39 smithi153 bash[35780]: cluster 2024-02-10T09:21:39.788132+0000 mgr.y (mgr.14140) 170 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:39 smithi153 bash[39117]: cluster 2024-02-10T09:21:39.788132+0000 mgr.y (mgr.14140) 170 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:39 smithi204 bash[35809]: cluster 2024-02-10T09:21:39.788132+0000 mgr.y (mgr.14140) 170 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:21:41.351 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":53,"fsid":"09b43244-c7f5-11ee-95b8-87774f69a715","created":"2024-02-10T09:16:42.107493+0000","modified":"2024-02-10T09:21:12.893128+0000","last_up_change":"2024-02-10T09:21:11.886623+0000","last_in_change":"2024-02-10T09:21:11.886623+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T09:16:51.902036+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8351789c-a971-44a1-aa2f-b16e87fb72a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":12,"up_from":14,"up_thru":51,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6803","nonce":2774397228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6805","nonce":2774397228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6809","nonce":2774397228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2774397228},{"type":"v1","addr":"172.21.15.153:6807","nonce":2774397228}]},"public_addr":"172.21.15.153:6803/2774397228","cluster_addr":"172.21.15.153:6805/2774397228","heartbeat_back_addr":"172.21.15.153:6809/2774397228","heartbeat_front_addr":"172.21.15.153:6807/2774397228","state":["exists","up"]},{"osd":1,"uuid":"fa508d13-ae9b-4d62-b567-1be1a9d46d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":20,"up_from":23,"up_thru":37,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6811","nonce":1003292780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6813","nonce":1003292780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6817","nonce":1003292780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1003292780},{"type":"v1","addr":"172.21.15.153:6815","nonce":1003292780}]},"public_addr":"172.21.15.153:6811/1003292780","cluster_addr":"172.21.15.153:6813/1003292780","heartbeat_back_addr":"172.21.15.153:6817/1003292780","heartbeat_front_addr":"172.21.15.153:6815/1003292780","state":["exists","up"]},{"osd":2,"uuid":"b6e5a960-d7e8-4e2c-b913-6848f7f0de36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6819","nonce":2495526904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6821","nonce":2495526904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6825","nonce":2495526904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2495526904},{"type":"v1","addr":"172.21.15.153:6823","nonce":2495526904}]},"public_addr":"172.21.15.153:6819/2495526904","cluster_addr":"172.21.15.153:6821/2495526904","heartbeat_back_addr":"172.21.15.153:6825/2495526904","heartbeat_front_addr":"172.21.15.153:6823/2495526904","state":["exists","up"]},{"osd":3,"uuid":"679b994c-2b1f-4f4d-9b26-64a3d365a094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6827","nonce":918480920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6829","nonce":918480920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6833","nonce":918480920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":918480920},{"type":"v1","addr":"172.21.15.153:6831","nonce":918480920}]},"public_addr":"172.21.15.153:6827/918480920","cluster_addr":"172.21.15.153:6829/918480920","heartbeat_back_addr":"172.21.15.153:6833/918480920","heartbeat_front_addr":"172.21.15.153:6831/918480920","state":["exists","up"]},{"osd":4,"uuid":"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6801","nonce":1147358970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6803","nonce":1147358970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6807","nonce":1147358970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1147358970},{"type":"v1","addr":"172.21.15.204:6805","nonce":1147358970}]},"public_addr":"172.21.15.204:6801/1147358970","cluster_addr":"172.21.15.204:6803/1147358970","heartbeat_back_addr":"172.21.15.204:6807/1147358970","heartbeat_front_addr":"172.21.15.204:6805/1147358970","state":["exists","up"]},{"osd":5,"uuid":"3ed1e885-ba58-4d66-a134-e63d325d4fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6809","nonce":4030840539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6811","nonce":4030840539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6815","nonce":4030840539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":4030840539},{"type":"v1","addr":"172.21.15.204:6813","nonce":4030840539}]},"public_addr":"172.21.15.204:6809/4030840539","cluster_addr":"172.21.15.204:6811/4030840539","heartbeat_back_addr":"172.21.15.204:6815/4030840539","heartbeat_front_addr":"172.21.15.204:6813/4030840539","state":["exists","up"]},{"osd":6,"uuid":"2f3e1b43-d7dd-44b2-be39-d240e518a51c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6817","nonce":2623910292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6819","nonce":2623910292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6823","nonce":2623910292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":2623910292},{"type":"v1","addr":"172.21.15.204:6821","nonce":2623910292}]},"public_addr":"172.21.15.204:6817/2623910292","cluster_addr":"172.21.15.204:6819/2623910292","heartbeat_back_addr":"172.21.15.204:6823/2623910292","heartbeat_front_addr":"172.21.15.204:6821/2623910292","state":["exists","up"]},{"osd":7,"uuid":"d38f33ce-8c0b-4235-be4f-4169e07e4cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6825","nonce":971867370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6827","nonce":971867370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6831","nonce":971867370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":971867370},{"type":"v1","addr":"172.21.15.204:6829","nonce":971867370}]},"public_addr":"172.21.15.204:6825/971867370","cluster_addr":"172.21.15.204:6827/971867370","heartbeat_back_addr":"172.21.15.204:6831/971867370","heartbeat_front_addr":"172.21.15.204:6829/971867370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-02-10T09:19:16.704450+0000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:18:58.903499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:19:22.123354+0000","dead_epoch":21},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:19:41.394864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:00.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:14.414485+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:33.299242+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:20:52.131341+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T09:21:10.437350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.153:0/2708898466":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/1486748846":"2024-02-11T09:17:20.528659+0000","172.21.15.153:6801/1546421638":"2024-02-11T09:17:20.528659+0000","172.21.15.153:6800/1546421638":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/94314011":"2024-02-11T09:17:20.528659+0000","172.21.15.153:0/1500108103":"2024-02-11T09:16:59.319422+0000","172.21.15.153:0/782778237":"2024-02-11T09:16:59.319422+0000","172.21.15.153:6800/1824985332":"2024-02-11T09:16:59.319422+0000","172.21.15.153:0/1431131370":"2024-02-11T09:16:59.319422+0000","172.21.15.153:6801/1824985332":"2024-02-11T09:16:59.319422+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2024-02-10T09:21:41.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:41 smithi153 bash[35780]: audit 2024-02-10T09:21:41.350448+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.153:0/845967337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:41.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:41 smithi153 bash[39117]: audit 2024-02-10T09:21:41.350448+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.153:0/845967337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:41.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:41 smithi204 bash[35809]: audit 2024-02-10T09:21:41.350448+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.153:0/845967337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T09:21:41.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T09:21:42.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:42 smithi153 bash[35780]: cluster 2024-02-10T09:21:41.788367+0000 mgr.y (mgr.14140) 171 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:42.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:42 smithi153 bash[39117]: cluster 2024-02-10T09:21:41.788367+0000 mgr.y (mgr.14140) 171 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:42 smithi204 bash[35809]: cluster 2024-02-10T09:21:41.788367+0000 mgr.y (mgr.14140) 171 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:44.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:43 smithi153 bash[39117]: cluster 2024-02-10T09:21:43.788603+0000 mgr.y (mgr.14140) 172 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:44.109 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:43 smithi153 bash[35780]: cluster 2024-02-10T09:21:43.788603+0000 mgr.y (mgr.14140) 172 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:43 smithi204 bash[35809]: cluster 2024-02-10T09:21:43.788603+0000 mgr.y (mgr.14140) 172 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:44.475 INFO:teuthology.orchestra.run.smithi153.stdout:60129542175 2024-02-10T09:21:44.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T09:21:46.077 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:45 smithi153 bash[39117]: cluster 2024-02-10T09:21:45.788865+0000 mgr.y (mgr.14140) 173 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:46.077 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:45 smithi153 bash[35780]: cluster 2024-02-10T09:21:45.788865+0000 mgr.y (mgr.14140) 173 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:45 smithi204 bash[35809]: cluster 2024-02-10T09:21:45.788865+0000 mgr.y (mgr.14140) 173 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:47.046 INFO:teuthology.orchestra.run.smithi153.stdout:98784247835 2024-02-10T09:21:47.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T09:21:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:47 smithi153 bash[35780]: cluster 2024-02-10T09:21:47.789114+0000 mgr.y (mgr.14140) 174 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:47 smithi153 bash[39117]: cluster 2024-02-10T09:21:47.789114+0000 mgr.y (mgr.14140) 174 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:47 smithi204 bash[35809]: cluster 2024-02-10T09:21:47.789114+0000 mgr.y (mgr.14140) 174 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:49.643 INFO:teuthology.orchestra.run.smithi153.stdout:115964117021 2024-02-10T09:21:49.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T09:21:50.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:49 smithi153 bash[35780]: cluster 2024-02-10T09:21:49.789350+0000 mgr.y (mgr.14140) 175 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:49 smithi153 bash[39117]: cluster 2024-02-10T09:21:49.789350+0000 mgr.y (mgr.14140) 175 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:49 smithi204 bash[35809]: cluster 2024-02-10T09:21:49.789350+0000 mgr.y (mgr.14140) 175 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:52.280 INFO:teuthology.orchestra.run.smithi153.stdout:137438953497 2024-02-10T09:21:52.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T09:21:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:52 smithi153 bash[35780]: cluster 2024-02-10T09:21:51.789583+0000 mgr.y (mgr.14140) 176 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:52 smithi153 bash[39117]: cluster 2024-02-10T09:21:51.789583+0000 mgr.y (mgr.14140) 176 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:52 smithi204 bash[35809]: cluster 2024-02-10T09:21:51.789583+0000 mgr.y (mgr.14140) 176 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:53.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:53 smithi153 bash[39117]: cluster 2024-02-10T09:21:53.789825+0000 mgr.y (mgr.14140) 177 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:53.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:53 smithi153 bash[35780]: cluster 2024-02-10T09:21:53.789825+0000 mgr.y (mgr.14140) 177 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:53 smithi204 bash[35809]: cluster 2024-02-10T09:21:53.789825+0000 mgr.y (mgr.14140) 177 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:54.817 INFO:teuthology.orchestra.run.smithi153.stdout:158913789974 2024-02-10T09:21:54.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T09:21:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:55 smithi153 bash[35780]: cluster 2024-02-10T09:21:55.790062+0000 mgr.y (mgr.14140) 178 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:55 smithi153 bash[39117]: cluster 2024-02-10T09:21:55.790062+0000 mgr.y (mgr.14140) 178 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:55 smithi204 bash[35809]: cluster 2024-02-10T09:21:55.790062+0000 mgr.y (mgr.14140) 178 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:57.353 INFO:teuthology.orchestra.run.smithi153.stdout:180388626451 2024-02-10T09:21:57.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T09:21:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:57 smithi153 bash[35780]: cluster 2024-02-10T09:21:57.790292+0000 mgr.y (mgr.14140) 179 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:57 smithi153 bash[39117]: cluster 2024-02-10T09:21:57.790292+0000 mgr.y (mgr.14140) 179 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:57 smithi204 bash[35809]: cluster 2024-02-10T09:21:57.790292+0000 mgr.y (mgr.14140) 179 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:59.624 INFO:teuthology.orchestra.run.smithi153.stdout:201863462927 2024-02-10T09:21:59.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T09:21:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:21:59 smithi153 bash[35780]: cluster 2024-02-10T09:21:59.790556+0000 mgr.y (mgr.14140) 180 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:21:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:21:59 smithi153 bash[39117]: cluster 2024-02-10T09:21:59.790556+0000 mgr.y (mgr.14140) 180 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:21:59 smithi204 bash[35809]: cluster 2024-02-10T09:21:59.790556+0000 mgr.y (mgr.14140) 180 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:02.071 INFO:teuthology.orchestra.run.smithi153.stdout:223338299404 2024-02-10T09:22:02.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T09:22:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:02 smithi153 bash[35780]: cluster 2024-02-10T09:22:01.790832+0000 mgr.y (mgr.14140) 181 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:02.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:02 smithi153 bash[39117]: cluster 2024-02-10T09:22:01.790832+0000 mgr.y (mgr.14140) 181 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:02 smithi204 bash[35809]: cluster 2024-02-10T09:22:01.790832+0000 mgr.y (mgr.14140) 181 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:03.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:03 smithi153 bash[39117]: cluster 2024-02-10T09:22:03.791112+0000 mgr.y (mgr.14140) 182 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:03.893 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:03 smithi153 bash[35780]: cluster 2024-02-10T09:22:03.791112+0000 mgr.y (mgr.14140) 182 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:04.251 INFO:teuthology.orchestra.run.smithi153.stdout:60129542179 2024-02-10T09:22:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:03 smithi204 bash[35809]: cluster 2024-02-10T09:22:03.791112+0000 mgr.y (mgr.14140) 182 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:04.807 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542179 for osd.0 2024-02-10T09:22:04.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T09:22:05.167 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:04 smithi153 bash[35780]: audit 2024-02-10T09:22:04.250898+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.153:0/2299455360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T09:22:05.168 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:04 smithi153 bash[39117]: audit 2024-02-10T09:22:04.250898+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.153:0/2299455360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T09:22:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:04 smithi204 bash[35809]: audit 2024-02-10T09:22:04.250898+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.153:0/2299455360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T09:22:06.070 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:05 smithi153 bash[35780]: cluster 2024-02-10T09:22:05.791366+0000 mgr.y (mgr.14140) 183 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:06.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:05 smithi153 bash[39117]: cluster 2024-02-10T09:22:05.791366+0000 mgr.y (mgr.14140) 183 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:05 smithi204 bash[35809]: cluster 2024-02-10T09:22:05.791366+0000 mgr.y (mgr.14140) 183 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:06.707 INFO:teuthology.orchestra.run.smithi153.stdout:98784247839 2024-02-10T09:22:07.302 INFO:tasks.cephadm.ceph_manager.ceph:need seq 98784247835 got 98784247839 for osd.1 2024-02-10T09:22:07.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T09:22:07.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:07 smithi153 bash[35780]: audit 2024-02-10T09:22:06.707515+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.153:0/1881572062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T09:22:07.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:07 smithi153 bash[39117]: audit 2024-02-10T09:22:06.707515+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.153:0/1881572062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T09:22:08.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:07 smithi204 bash[35809]: audit 2024-02-10T09:22:06.707515+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.153:0/1881572062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T09:22:08.814 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:08 smithi153 bash[35780]: cluster 2024-02-10T09:22:07.791609+0000 mgr.y (mgr.14140) 184 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:08.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:08 smithi153 bash[39117]: cluster 2024-02-10T09:22:07.791609+0000 mgr.y (mgr.14140) 184 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:09.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:08 smithi204 bash[35809]: cluster 2024-02-10T09:22:07.791609+0000 mgr.y (mgr.14140) 184 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:09.185 INFO:teuthology.orchestra.run.smithi153.stdout:115964117024 2024-02-10T09:22:09.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:09 smithi153 bash[35780]: audit 2024-02-10T09:22:09.185131+0000 mon.a (mon.0) 528 : audit [DBG] from='client.? 172.21.15.153:0/2480797133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T09:22:09.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:09 smithi153 bash[39117]: audit 2024-02-10T09:22:09.185131+0000 mon.a (mon.0) 528 : audit [DBG] from='client.? 172.21.15.153:0/2480797133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T09:22:10.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117021 got 115964117024 for osd.2 2024-02-10T09:22:10.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T09:22:10.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:09 smithi204 bash[35809]: audit 2024-02-10T09:22:09.185131+0000 mon.a (mon.0) 528 : audit [DBG] from='client.? 172.21.15.153:0/2480797133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T09:22:10.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:10 smithi153 bash[35780]: cluster 2024-02-10T09:22:09.791868+0000 mgr.y (mgr.14140) 185 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:10.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:10 smithi153 bash[39117]: cluster 2024-02-10T09:22:09.791868+0000 mgr.y (mgr.14140) 185 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:10 smithi204 bash[35809]: cluster 2024-02-10T09:22:09.791868+0000 mgr.y (mgr.14140) 185 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:12.074 INFO:teuthology.orchestra.run.smithi153.stdout:137438953501 2024-02-10T09:22:12.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:12 smithi153 bash[35780]: cluster 2024-02-10T09:22:11.792121+0000 mgr.y (mgr.14140) 186 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:12.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:12 smithi153 bash[35780]: audit 2024-02-10T09:22:12.073631+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.153:0/974075932' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T09:22:12.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:12 smithi153 bash[39117]: cluster 2024-02-10T09:22:11.792121+0000 mgr.y (mgr.14140) 186 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:12.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:12 smithi153 bash[39117]: audit 2024-02-10T09:22:12.073631+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.153:0/974075932' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T09:22:12.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953497 got 137438953501 for osd.3 2024-02-10T09:22:12.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T09:22:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:12 smithi204 bash[35809]: cluster 2024-02-10T09:22:11.792121+0000 mgr.y (mgr.14140) 186 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:12 smithi204 bash[35809]: audit 2024-02-10T09:22:12.073631+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.153:0/974075932' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T09:22:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:13 smithi153 bash[35780]: cluster 2024-02-10T09:22:13.792372+0000 mgr.y (mgr.14140) 187 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:13 smithi153 bash[39117]: cluster 2024-02-10T09:22:13.792372+0000 mgr.y (mgr.14140) 187 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:13 smithi204 bash[35809]: cluster 2024-02-10T09:22:13.792372+0000 mgr.y (mgr.14140) 187 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:14.754 INFO:teuthology.orchestra.run.smithi153.stdout:158913789978 2024-02-10T09:22:15.099 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:14 smithi153 bash[35780]: audit 2024-02-10T09:22:14.753634+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.153:0/468480799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T09:22:15.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:14 smithi153 bash[39117]: audit 2024-02-10T09:22:14.753634+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.153:0/468480799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T09:22:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:14 smithi204 bash[35809]: audit 2024-02-10T09:22:14.753634+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.153:0/468480799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T09:22:15.492 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789974 got 158913789978 for osd.4 2024-02-10T09:22:15.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T09:22:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:15 smithi153 bash[35780]: cluster 2024-02-10T09:22:15.792616+0000 mgr.y (mgr.14140) 188 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:15 smithi153 bash[39117]: cluster 2024-02-10T09:22:15.792616+0000 mgr.y (mgr.14140) 188 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:15 smithi204 bash[35809]: cluster 2024-02-10T09:22:15.792616+0000 mgr.y (mgr.14140) 188 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:17.373 INFO:teuthology.orchestra.run.smithi153.stdout:180388626455 2024-02-10T09:22:17.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:17 smithi153 bash[35780]: audit 2024-02-10T09:22:17.372843+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.153:0/3087242998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T09:22:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:17 smithi153 bash[39117]: audit 2024-02-10T09:22:17.372843+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.153:0/3087242998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T09:22:18.027 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626451 got 180388626455 for osd.5 2024-02-10T09:22:18.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T09:22:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:17 smithi204 bash[35809]: audit 2024-02-10T09:22:17.372843+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.153:0/3087242998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T09:22:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:18 smithi153 bash[35780]: cluster 2024-02-10T09:22:17.792866+0000 mgr.y (mgr.14140) 189 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:18 smithi153 bash[39117]: cluster 2024-02-10T09:22:17.792866+0000 mgr.y (mgr.14140) 189 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:18 smithi204 bash[35809]: cluster 2024-02-10T09:22:17.792866+0000 mgr.y (mgr.14140) 189 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:19.946 INFO:teuthology.orchestra.run.smithi153.stdout:201863462931 2024-02-10T09:22:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:19 smithi153 bash[35780]: cluster 2024-02-10T09:22:19.793158+0000 mgr.y (mgr.14140) 190 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:19 smithi153 bash[39117]: cluster 2024-02-10T09:22:19.793158+0000 mgr.y (mgr.14140) 190 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:19 smithi204 bash[35809]: cluster 2024-02-10T09:22:19.793158+0000 mgr.y (mgr.14140) 190 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:20.575 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462927 got 201863462931 for osd.6 2024-02-10T09:22:20.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T09:22:20.888 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:20 smithi153 bash[35780]: audit 2024-02-10T09:22:19.945947+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? 172.21.15.153:0/2173476624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T09:22:21.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:20 smithi153 bash[39117]: audit 2024-02-10T09:22:19.945947+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? 172.21.15.153:0/2173476624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T09:22:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:20 smithi204 bash[35809]: audit 2024-02-10T09:22:19.945947+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? 172.21.15.153:0/2173476624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T09:22:22.577 INFO:teuthology.orchestra.run.smithi153.stdout:223338299408 2024-02-10T09:22:22.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:22 smithi153 bash[39117]: cluster 2024-02-10T09:22:21.793419+0000 mgr.y (mgr.14140) 191 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:22.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:22 smithi153 bash[39117]: audit 2024-02-10T09:22:22.577252+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.21.15.153:0/697801829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T09:22:22.920 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:22 smithi153 bash[35780]: cluster 2024-02-10T09:22:21.793419+0000 mgr.y (mgr.14140) 191 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:22.920 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:22 smithi153 bash[35780]: audit 2024-02-10T09:22:22.577252+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.21.15.153:0/697801829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T09:22:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:22 smithi204 bash[35809]: cluster 2024-02-10T09:22:21.793419+0000 mgr.y (mgr.14140) 191 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:22 smithi204 bash[35809]: audit 2024-02-10T09:22:22.577252+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.21.15.153:0/697801829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T09:22:23.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299404 got 223338299408 for osd.7 2024-02-10T09:22:23.305 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T09:22:23.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T09:22:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:23 smithi153 bash[35780]: cluster 2024-02-10T09:22:23.793666+0000 mgr.y (mgr.14140) 192 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:23 smithi153 bash[39117]: cluster 2024-02-10T09:22:23.793666+0000 mgr.y (mgr.14140) 192 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:23 smithi204 bash[35809]: cluster 2024-02-10T09:22:23.793666+0000 mgr.y (mgr.14140) 192 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:25.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:22:25.119 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-02-10T09:22:25.891 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-02-10T09:22:23.793552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":8392600,"kb_used_data":3928,"kb_used_omap":13,"kb_used_meta":8388594,"kb_avail":741568616,"statfs":{"total":767960285184,"available":759366262784,"internally_reserved":8589934592,"allocated":4022272,"data_stored":1586992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13351,"internal_metadata":8589921241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"1.0","version":"20'1","reported_seq":"14","reported_epoch":"53","state":"active+clean","last_fresh":"2024-02-10T09:21:12.905156+0000","last_change":"2024-02-10T09:21:12.905156+0000","last_active":"2024-02-10T09:21:12.905156+0000","last_peered":"2024-02-10T09:21:12.905156+0000","last_clean":"2024-02-10T09:21:12.905156+0000","last_became_active":"2024-02-10T09:21:12.902590+0000","last_became_peered":"2024-02-10T09:21:12.902590+0000","last_unstale":"2024-02-10T09:21:12.905156+0000","last_undegraded":"2024-02-10T09:21:12.905156+0000","last_fullsized":"2024-02-10T09:21:12.905156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":2,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T09:16:51.902061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T09:16:51.902061+0000","last_clean_scrub_stamp":"2024-02-10T09:16:51.902061+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8228,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299409,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.378}]},{"osd":1,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.311},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41599999999999998}]},{"osd":5,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.34200000000000003}]},{"osd":6,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.35499999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.63},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.56699999999999995}]},{"osd":3,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.61199999999999999}]},{"osd":4,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.443}]},{"osd":5,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.52300000000000002}]}]},{"osd":1,"up_from":23,"seq":98784247843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049072,"kb_used_data":488,"kb_used_omap":8,"kb_used_meta":1048567,"kb_avail":92696080,"statfs":{"total":95995035648,"available":94920785920,"internally_reserved":1073741824,"allocated":499712,"data_stored":198136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8228,"internal_metadata":1073733596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.32400000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33100000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Feb 10 09:21:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.34000000000000002,"15min":0.33600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.35199999999999998,"15min":0.34899999999999998},"min":{"1min":0.246,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.315}]},{"osd":3,"last update":"Sat Feb 10 09:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39300000000000002,"15min":0.40000000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.502,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.27900000000000003}]},{"osd":4,"last update":"Sat Feb 10 09:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.626,"15min":0.64100000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74099999999999999,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.62,"15min":0.63600000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69599999999999995,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.503}]},{"osd":5,"last update":"Sat Feb 10 09:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.496},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Feb 10 09:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.58199999999999996}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.48899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":5,"kb_used_meta":1048570,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5123,"internal_metadata":1073736701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 09:21:44 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.34200000000000003,"15min":0.33700000000000002},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.253}]},{"osd":2,"last update":"Sat Feb 10 09:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.253,"5min":0.216,"15min":0.216},"max":{"1min":0.52100000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.311}]},{"osd":3,"last update":"Sat Feb 10 09:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.40899999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Feb 10 09:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50700000000000001,"15min":0.503},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65200000000000002}]},{"osd":5,"last update":"Sat Feb 10 09:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.67100000000000004}]},{"osd":6,"last update":"Sat Feb 10 09:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.70399999999999996}]},{"osd":7,"last update":"Sat Feb 10 09:22:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002}]}]},{"osd":2,"up_from":27,"seq":115964117028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.33800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.442}]},{"osd":1,"last update":"Sat Feb 10 09:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34000000000000002,"15min":0.33800000000000002},"min":{"1min":0.224,"5min":0.218,"15min":0.218},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.372,"15min":0.36699999999999999},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.375}]},{"osd":3,"last update":"Sat Feb 10 09:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Feb 10 09:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.35999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Feb 10 09:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.499}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59199999999999997}]}]},{"osd":3,"up_from":32,"seq":137438953504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28899999999999998}]},{"osd":1,"last update":"Sat Feb 10 09:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.373,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.315},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.253}]},{"osd":2,"last update":"Sat Feb 10 09:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40500000000000003,"15min":0.41099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39000000000000001,"15min":0.39600000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.29999999999999999}]},{"osd":4,"last update":"Sat Feb 10 09:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.36699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.629},{"interface":"front","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45100000000000001}]},{"osd":5,"last update":"Sat Feb 10 09:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Feb 10 09:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.57199999999999995}]}]},{"osd":4,"up_from":37,"seq":158913789980,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.505,"15min":0.505},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.64400000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.64400000000000002}]},{"osd":1,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51500000000000001,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.61899999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.61399999999999999,"5min":0.626,"15min":0.626},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52300000000000002,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.622,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.44600000000000001}]},{"osd":3,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.71899999999999997}]},{"osd":5,"last update":"Sat Feb 10 09:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.5}]},{"osd":6,"last update":"Sat Feb 10 09:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.626},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.67200000000000004}]},{"osd":7,"last update":"Sat Feb 10 09:22:12 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.65900000000000003}]}]},{"osd":5,"up_from":42,"seq":180388626456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049072,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696080,"statfs":{"total":95995035648,"available":94920785920,"internally_reserved":1073741824,"allocated":499712,"data_stored":198156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.505}]},{"osd":3,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.442},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.39700000000000002}]},{"osd":6,"last update":"Sat Feb 10 09:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.435}]},{"osd":7,"last update":"Sat Feb 10 09:22:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.52700000000000002}]}]}]}} 2024-02-10T09:22:25.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T09:22:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:25 smithi153 bash[35780]: audit 2024-02-10T09:22:25.115180+0000 mgr.y (mgr.14140) 193 : audit [DBG] from='client.24490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:25 smithi153 bash[35780]: cluster 2024-02-10T09:22:25.793883+0000 mgr.y (mgr.14140) 194 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:26.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:25 smithi153 bash[39117]: audit 2024-02-10T09:22:25.115180+0000 mgr.y (mgr.14140) 193 : audit [DBG] from='client.24490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:26.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:25 smithi153 bash[39117]: cluster 2024-02-10T09:22:25.793883+0000 mgr.y (mgr.14140) 194 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:25 smithi204 bash[35809]: audit 2024-02-10T09:22:25.115180+0000 mgr.y (mgr.14140) 193 : audit [DBG] from='client.24490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:25 smithi204 bash[35809]: cluster 2024-02-10T09:22:25.793883+0000 mgr.y (mgr.14140) 194 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:27.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:22:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-02-10T09:22:28.169 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:27 smithi153 bash[39117]: cluster 2024-02-10T09:22:27.794144+0000 mgr.y (mgr.14140) 195 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:28.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:27 smithi153 bash[35780]: cluster 2024-02-10T09:22:27.794144+0000 mgr.y (mgr.14140) 195 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:27 smithi204 bash[35809]: cluster 2024-02-10T09:22:27.794144+0000 mgr.y (mgr.14140) 195 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:28.506 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-02-10T09:22:27.794030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":8392600,"kb_used_data":3928,"kb_used_omap":13,"kb_used_meta":8388594,"kb_avail":741568616,"statfs":{"total":767960285184,"available":759366262784,"internally_reserved":8589934592,"allocated":4022272,"data_stored":1586992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13351,"internal_metadata":8589921241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001523"},"pg_stats":[{"pgid":"1.0","version":"20'1","reported_seq":"14","reported_epoch":"53","state":"active+clean","last_fresh":"2024-02-10T09:21:12.905156+0000","last_change":"2024-02-10T09:21:12.905156+0000","last_active":"2024-02-10T09:21:12.905156+0000","last_peered":"2024-02-10T09:21:12.905156+0000","last_clean":"2024-02-10T09:21:12.905156+0000","last_became_active":"2024-02-10T09:21:12.902590+0000","last_became_peered":"2024-02-10T09:21:12.902590+0000","last_unstale":"2024-02-10T09:21:12.905156+0000","last_undegraded":"2024-02-10T09:21:12.905156+0000","last_fullsized":"2024-02-10T09:21:12.905156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":2,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T09:16:51.902061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T09:16:51.902061+0000","last_clean_scrub_stamp":"2024-02-10T09:16:51.902061+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8228,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299409,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.378}]},{"osd":1,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.311},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41599999999999998}]},{"osd":5,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.34200000000000003}]},{"osd":6,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.35499999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.378}]},{"osd":2,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.499},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.434}]},{"osd":4,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.40600000000000003}]},{"osd":5,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48399999999999999}]}]},{"osd":1,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049072,"kb_used_data":488,"kb_used_omap":8,"kb_used_meta":1048567,"kb_avail":92696080,"statfs":{"total":95995035648,"available":94920785920,"internally_reserved":1073741824,"allocated":499712,"data_stored":198136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8228,"internal_metadata":1073733596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.32400000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.254},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33100000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.26300000000000001}]},{"osd":2,"last update":"Sat Feb 10 09:21:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.34000000000000002,"15min":0.33600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.35199999999999998,"15min":0.34899999999999998},"min":{"1min":0.246,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.32200000000000001}]},{"osd":3,"last update":"Sat Feb 10 09:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39300000000000002,"15min":0.40000000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.502,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.317},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.38200000000000001}]},{"osd":4,"last update":"Sat Feb 10 09:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.626,"15min":0.64100000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74099999999999999,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.62,"15min":0.63600000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69599999999999995,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Feb 10 09:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.62}]},{"osd":6,"last update":"Sat Feb 10 09:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542184,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":5,"kb_used_meta":1048570,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5123,"internal_metadata":1073736701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 09:21:44 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.34200000000000003,"15min":0.33700000000000002},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.253}]},{"osd":2,"last update":"Sat Feb 10 09:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.253,"5min":0.216,"15min":0.216},"max":{"1min":0.52100000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.311}]},{"osd":3,"last update":"Sat Feb 10 09:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.40899999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Feb 10 09:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50700000000000001,"15min":0.503},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65200000000000002}]},{"osd":5,"last update":"Sat Feb 10 09:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.67100000000000004}]},{"osd":6,"last update":"Sat Feb 10 09:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.70399999999999996}]},{"osd":7,"last update":"Sat Feb 10 09:22:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002}]}]},{"osd":2,"up_from":27,"seq":115964117028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.33800000000000002},"min":{"1min":0.23699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.442}]},{"osd":1,"last update":"Sat Feb 10 09:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34000000000000002,"15min":0.33800000000000002},"min":{"1min":0.224,"5min":0.218,"15min":0.218},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.372,"15min":0.36699999999999999},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.375}]},{"osd":3,"last update":"Sat Feb 10 09:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Feb 10 09:22:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.35999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Feb 10 09:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Feb 10 09:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.499}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59199999999999997}]}]},{"osd":3,"up_from":32,"seq":137438953504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28899999999999998}]},{"osd":1,"last update":"Sat Feb 10 09:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.373,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.315},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.253}]},{"osd":2,"last update":"Sat Feb 10 09:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40500000000000003,"15min":0.41099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39000000000000001,"15min":0.39600000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.29999999999999999}]},{"osd":4,"last update":"Sat Feb 10 09:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.36699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.629},{"interface":"front","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45100000000000001}]},{"osd":5,"last update":"Sat Feb 10 09:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Feb 10 09:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Feb 10 09:22:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.57199999999999995}]}]},{"osd":4,"up_from":37,"seq":158913789981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049076,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696076,"statfs":{"total":95995035648,"available":94920781824,"internally_reserved":1073741824,"allocated":503808,"data_stored":198450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.505,"15min":0.505},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.64400000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.52900000000000003}]},{"osd":1,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51500000000000001,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.61899999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.61399999999999999,"5min":0.626,"15min":0.626},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52300000000000002,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.622,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.55200000000000005}]},{"osd":3,"last update":"Sat Feb 10 09:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Feb 10 09:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.34100000000000003}]},{"osd":6,"last update":"Sat Feb 10 09:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40500000000000003}]},{"osd":7,"last update":"Sat Feb 10 09:22:12 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.435}]}]},{"osd":5,"up_from":42,"seq":180388626457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1049072,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":92696080,"statfs":{"total":95995035648,"available":94920785920,"internally_reserved":1073741824,"allocated":499712,"data_stored":198156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.47099999999999997}]},{"osd":1,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.442},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.436}]},{"osd":2,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42599999999999999}]},{"osd":3,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Feb 10 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.375}]},{"osd":6,"last update":"Sat Feb 10 09:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.312}]},{"osd":7,"last update":"Sat Feb 10 09:22:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.38800000000000001}]}]}]}} 2024-02-10T09:22:28.508 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T09:22:28.508 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T09:22:28.508 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T09:22:28.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T09:22:29.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:28 smithi153 bash[35780]: audit 2024-02-10T09:22:27.851188+0000 mgr.y (mgr.14140) 196 : audit [DBG] from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:28 smithi153 bash[39117]: audit 2024-02-10T09:22:27.851188+0000 mgr.y (mgr.14140) 196 : audit [DBG] from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:28 smithi204 bash[35809]: audit 2024-02-10T09:22:27.851188+0000 mgr.y (mgr.14140) 196 : audit [DBG] from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:30.014 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:29 smithi153 bash[39117]: cluster 2024-02-10T09:22:29.794385+0000 mgr.y (mgr.14140) 197 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:30.015 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:29 smithi153 bash[35780]: cluster 2024-02-10T09:22:29.794385+0000 mgr.y (mgr.14140) 197 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:29 smithi204 bash[35809]: cluster 2024-02-10T09:22:29.794385+0000 mgr.y (mgr.14140) 197 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:30.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:22:30.892 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T09:22:30.892 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T09:22:30.893 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T09:22:30.893 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T09:22:30.903 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi153.front.sepia.ceph.com 2024-02-10T09:22:30.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-02-10T09:22:30.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:30 smithi153 bash[39117]: audit 2024-02-10T09:22:30.370480+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.153:0/1788710758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T09:22:30.931 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:30 smithi153 bash[35780]: audit 2024-02-10T09:22:30.370480+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.153:0/1788710758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T09:22:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:30 smithi204 bash[35809]: audit 2024-02-10T09:22:30.370480+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.153:0/1788710758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T09:22:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:32 smithi204 bash[35809]: cluster 2024-02-10T09:22:31.794638+0000 mgr.y (mgr.14140) 198 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:33.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:32 smithi153 bash[35780]: cluster 2024-02-10T09:22:31.794638+0000 mgr.y (mgr.14140) 198 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:33.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:32 smithi153 bash[39117]: cluster 2024-02-10T09:22:31.794638+0000 mgr.y (mgr.14140) 198 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:33.874 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:22:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "8a2fe70c-c3a7-4c18-b934-bbe49646e5fb", 2024-02-10T09:22:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "r", 2024-02-10T09:22:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: "current_period": "533d482e-e080-4bd3-9fb0-5cc676d83a48", 2024-02-10T09:22:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1 2024-02-10T09:22:33.874 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:22:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:33 smithi153 bash[35780]: cluster 2024-02-10T09:22:32.834337+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-10T09:22:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:33 smithi153 bash[35780]: audit 2024-02-10T09:22:32.835488+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.153:0/3022615978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T09:22:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:33 smithi153 bash[35780]: audit 2024-02-10T09:22:32.835901+0000 mon.a (mon.0) 532 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T09:22:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:33 smithi153 bash[39117]: cluster 2024-02-10T09:22:32.834337+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-10T09:22:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:33 smithi153 bash[39117]: audit 2024-02-10T09:22:32.835488+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.153:0/3022615978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T09:22:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:33 smithi153 bash[39117]: audit 2024-02-10T09:22:32.835901+0000 mon.a (mon.0) 532 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T09:22:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:33 smithi204 bash[35809]: cluster 2024-02-10T09:22:32.834337+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-10T09:22:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:33 smithi204 bash[35809]: audit 2024-02-10T09:22:32.835488+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.153:0/3022615978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T09:22:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:33 smithi204 bash[35809]: audit 2024-02-10T09:22:32.835901+0000 mon.a (mon.0) 532 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T09:22:34.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-02-10T09:22:35.065 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:34 smithi153 bash[35780]: cluster 2024-02-10T09:22:33.794958+0000 mgr.y (mgr.14140) 199 : cluster [DBG] pgmap v199: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:35.065 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:34 smithi153 bash[35780]: audit 2024-02-10T09:22:33.839951+0000 mon.a (mon.0) 533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T09:22:35.065 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:34 smithi153 bash[35780]: cluster 2024-02-10T09:22:33.840158+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-10T09:22:35.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:34 smithi153 bash[39117]: cluster 2024-02-10T09:22:33.794958+0000 mgr.y (mgr.14140) 199 : cluster [DBG] pgmap v199: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:35.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:34 smithi153 bash[39117]: audit 2024-02-10T09:22:33.839951+0000 mon.a (mon.0) 533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T09:22:35.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:34 smithi153 bash[39117]: cluster 2024-02-10T09:22:33.840158+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-10T09:22:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:34 smithi204 bash[35809]: cluster 2024-02-10T09:22:33.794958+0000 mgr.y (mgr.14140) 199 : cluster [DBG] pgmap v199: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:22:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:34 smithi204 bash[35809]: audit 2024-02-10T09:22:33.839951+0000 mon.a (mon.0) 533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T09:22:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:34 smithi204 bash[35809]: cluster 2024-02-10T09:22:33.840158+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-10T09:22:36.182 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:35 smithi153 bash[39117]: cluster 2024-02-10T09:22:34.844822+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-10T09:22:36.182 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:35 smithi153 bash[39117]: cluster 2024-02-10T09:22:35.795470+0000 mgr.y (mgr.14140) 200 : cluster [DBG] pgmap v202: 33 pgs: 13 creating+peering, 8 unknown, 12 active+clean; 46 B data, 4.0 MiB used, 707 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-10T09:22:36.182 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:35 smithi153 bash[35780]: cluster 2024-02-10T09:22:34.844822+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-10T09:22:36.183 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:35 smithi153 bash[35780]: cluster 2024-02-10T09:22:35.795470+0000 mgr.y (mgr.14140) 200 : cluster [DBG] pgmap v202: 33 pgs: 13 creating+peering, 8 unknown, 12 active+clean; 46 B data, 4.0 MiB used, 707 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-10T09:22:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:35 smithi204 bash[35809]: cluster 2024-02-10T09:22:34.844822+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-10T09:22:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:35 smithi204 bash[35809]: cluster 2024-02-10T09:22:35.795470+0000 mgr.y (mgr.14140) 200 : cluster [DBG] pgmap v202: 33 pgs: 13 creating+peering, 8 unknown, 12 active+clean; 46 B data, 4.0 MiB used, 707 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-10T09:22:36.440 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:22:36.440 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "b5782749-01e9-4840-af19-5fe16b02c924", 2024-02-10T09:22:36.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "default", 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "api_name": "default", 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "is_master": "true", 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "endpoints": [], 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "hostnames": [], 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "hostnames_s3website": [], 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "master_zone": "", 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "zones": [], 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "placement_targets": [], 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "default_placement": "", 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "realm_id": "8a2fe70c-c3a7-4c18-b934-bbe49646e5fb", 2024-02-10T09:22:36.441 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_policy": { 2024-02-10T09:22:36.442 INFO:teuthology.orchestra.run.smithi153.stdout: "groups": [] 2024-02-10T09:22:36.442 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:36.442 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:22:37.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-02-10T09:22:38.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:37 smithi153 bash[39117]: cluster 2024-02-10T09:22:37.795836+0000 mgr.y (mgr.14140) 201 : cluster [DBG] pgmap v203: 33 pgs: 13 creating+peering, 20 active+clean; 193 B data, 4.1 MiB used, 707 GiB / 715 GiB avail; 170 B/s rd, 853 B/s wr, 1 op/s 2024-02-10T09:22:38.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:37 smithi153 bash[35780]: cluster 2024-02-10T09:22:37.795836+0000 mgr.y (mgr.14140) 201 : cluster [DBG] pgmap v203: 33 pgs: 13 creating+peering, 20 active+clean; 193 B data, 4.1 MiB used, 707 GiB / 715 GiB avail; 170 B/s rd, 853 B/s wr, 1 op/s 2024-02-10T09:22:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:37 smithi204 bash[35809]: cluster 2024-02-10T09:22:37.795836+0000 mgr.y (mgr.14140) 201 : cluster [DBG] pgmap v203: 33 pgs: 13 creating+peering, 20 active+clean; 193 B data, 4.1 MiB used, 707 GiB / 715 GiB avail; 170 B/s rd, 853 B/s wr, 1 op/s 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "51ae5349-887c-4d78-95df-1d0f5a7e9eb5", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "z", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "domain_root": "z.rgw.meta:root", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "control_pool": "z.rgw.control", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "gc_pool": "z.rgw.log:gc", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "lc_pool": "z.rgw.log:lc", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "log_pool": "z.rgw.log", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "roles_pool": "z.rgw.meta:roles", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-02-10T09:22:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "otp_pool": "z.rgw.otp", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "system_key": { 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "access_key": "", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "secret_key": "" 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "placement_pools": [ 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "key": "default-placement", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "val": { 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "index_pool": "z.rgw.buckets.index", 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "storage_classes": { 2024-02-10T09:22:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: "STANDARD": { 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: "data_pool": "z.rgw.buckets.data" 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: "index_type": 0 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout: "realm_id": "8a2fe70c-c3a7-4c18-b934-bbe49646e5fb" 2024-02-10T09:22:38.872 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:22:39.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-02-10T09:22:40.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:39 smithi153 bash[35780]: cluster 2024-02-10T09:22:39.796156+0000 mgr.y (mgr.14140) 202 : cluster [DBG] pgmap v204: 33 pgs: 33 active+clean; 562 B data, 4.2 MiB used, 707 GiB / 715 GiB avail; 294 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-02-10T09:22:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:39 smithi153 bash[39117]: cluster 2024-02-10T09:22:39.796156+0000 mgr.y (mgr.14140) 202 : cluster [DBG] pgmap v204: 33 pgs: 33 active+clean; 562 B data, 4.2 MiB used, 707 GiB / 715 GiB avail; 294 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-02-10T09:22:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:39 smithi204 bash[35809]: cluster 2024-02-10T09:22:39.796156+0000 mgr.y (mgr.14140) 202 : cluster [DBG] pgmap v204: 33 pgs: 33 active+clean; 562 B data, 4.2 MiB used, 707 GiB / 715 GiB avail; 294 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-02-10T09:22:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:42 smithi153 bash[35780]: cluster 2024-02-10T09:22:41.408845+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-10T09:22:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:42 smithi153 bash[35780]: audit 2024-02-10T09:22:41.412404+0000 mon.a (mon.0) 537 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T09:22:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:42 smithi153 bash[35780]: cluster 2024-02-10T09:22:41.796404+0000 mgr.y (mgr.14140) 203 : cluster [DBG] pgmap v206: 65 pgs: 32 unknown, 33 active+clean; 996 B data, 4.3 MiB used, 707 GiB / 715 GiB avail; 514 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T09:22:42.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:42 smithi153 bash[39117]: cluster 2024-02-10T09:22:41.408845+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-10T09:22:42.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:42 smithi153 bash[39117]: audit 2024-02-10T09:22:41.412404+0000 mon.a (mon.0) 537 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T09:22:42.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:42 smithi153 bash[39117]: cluster 2024-02-10T09:22:41.796404+0000 mgr.y (mgr.14140) 203 : cluster [DBG] pgmap v206: 65 pgs: 32 unknown, 33 active+clean; 996 B data, 4.3 MiB used, 707 GiB / 715 GiB avail; 514 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T09:22:42.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:42 smithi204 bash[35809]: cluster 2024-02-10T09:22:41.408845+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-10T09:22:42.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:42 smithi204 bash[35809]: audit 2024-02-10T09:22:41.412404+0000 mon.a (mon.0) 537 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T09:22:42.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:42 smithi204 bash[35809]: cluster 2024-02-10T09:22:41.796404+0000 mgr.y (mgr.14140) 203 : cluster [DBG] pgmap v206: 65 pgs: 32 unknown, 33 active+clean; 996 B data, 4.3 MiB used, 707 GiB / 715 GiB avail; 514 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T09:22:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:43 smithi153 bash[35780]: audit 2024-02-10T09:22:42.410985+0000 mon.a (mon.0) 538 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T09:22:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:43 smithi153 bash[35780]: cluster 2024-02-10T09:22:42.411058+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-10T09:22:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:43 smithi153 bash[39117]: audit 2024-02-10T09:22:42.410985+0000 mon.a (mon.0) 538 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T09:22:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:43 smithi153 bash[39117]: cluster 2024-02-10T09:22:42.411058+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-10T09:22:43.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:43 smithi204 bash[35809]: audit 2024-02-10T09:22:42.410985+0000 mon.a (mon.0) 538 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T09:22:43.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:43 smithi204 bash[35809]: cluster 2024-02-10T09:22:42.411058+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-10T09:22:44.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:44 smithi204 bash[35809]: cluster 2024-02-10T09:22:43.422373+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-10T09:22:44.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:44 smithi204 bash[35809]: audit 2024-02-10T09:22:43.431410+0000 mon.a (mon.0) 541 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T09:22:44.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:44 smithi204 bash[35809]: cluster 2024-02-10T09:22:43.796774+0000 mgr.y (mgr.14140) 204 : cluster [DBG] pgmap v209: 97 pgs: 1 creating+peering, 48 unknown, 48 active+clean; 1.8 KiB data, 4.4 MiB used, 707 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T09:22:44.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:44 smithi204 bash[35809]: audit 2024-02-10T09:22:44.424088+0000 mon.a (mon.0) 542 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T09:22:44.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:44 smithi204 bash[35809]: cluster 2024-02-10T09:22:44.424196+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-10T09:22:44.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[39117]: cluster 2024-02-10T09:22:43.422373+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[39117]: audit 2024-02-10T09:22:43.431410+0000 mon.a (mon.0) 541 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[39117]: cluster 2024-02-10T09:22:43.796774+0000 mgr.y (mgr.14140) 204 : cluster [DBG] pgmap v209: 97 pgs: 1 creating+peering, 48 unknown, 48 active+clean; 1.8 KiB data, 4.4 MiB used, 707 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[39117]: audit 2024-02-10T09:22:44.424088+0000 mon.a (mon.0) 542 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[39117]: cluster 2024-02-10T09:22:44.424196+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[35780]: cluster 2024-02-10T09:22:43.422373+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[35780]: audit 2024-02-10T09:22:43.431410+0000 mon.a (mon.0) 541 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[35780]: cluster 2024-02-10T09:22:43.796774+0000 mgr.y (mgr.14140) 204 : cluster [DBG] pgmap v209: 97 pgs: 1 creating+peering, 48 unknown, 48 active+clean; 1.8 KiB data, 4.4 MiB used, 707 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[35780]: audit 2024-02-10T09:22:44.424088+0000 mon.a (mon.0) 542 : audit [INF] from='client.? 172.21.15.153:0/3994404756' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T09:22:44.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:44 smithi153 bash[35780]: cluster 2024-02-10T09:22:44.424196+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[39117]: cluster 2024-02-10T09:22:45.425777+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[39117]: audit 2024-02-10T09:22:45.430976+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[39117]: audit 2024-02-10T09:22:45.431978+0000 mon.a (mon.0) 545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[39117]: cluster 2024-02-10T09:22:45.797290+0000 mgr.y (mgr.14140) 205 : cluster [DBG] pgmap v212: 129 pgs: 5 creating+peering, 51 unknown, 73 active+clean; 1.8 KiB data, 4.5 MiB used, 707 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[35780]: cluster 2024-02-10T09:22:45.425777+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[35780]: audit 2024-02-10T09:22:45.430976+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T09:22:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[35780]: audit 2024-02-10T09:22:45.431978+0000 mon.a (mon.0) 545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T09:22:46.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:46 smithi153 bash[35780]: cluster 2024-02-10T09:22:45.797290+0000 mgr.y (mgr.14140) 205 : cluster [DBG] pgmap v212: 129 pgs: 5 creating+peering, 51 unknown, 73 active+clean; 1.8 KiB data, 4.5 MiB used, 707 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T09:22:46.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:46 smithi204 bash[35809]: cluster 2024-02-10T09:22:45.425777+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-10T09:22:46.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:46 smithi204 bash[35809]: audit 2024-02-10T09:22:45.430976+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T09:22:46.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:46 smithi204 bash[35809]: audit 2024-02-10T09:22:45.431978+0000 mon.a (mon.0) 545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T09:22:46.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:46 smithi204 bash[35809]: cluster 2024-02-10T09:22:45.797290+0000 mgr.y (mgr.14140) 205 : cluster [DBG] pgmap v212: 129 pgs: 5 creating+peering, 51 unknown, 73 active+clean; 1.8 KiB data, 4.5 MiB used, 707 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[35780]: audit 2024-02-10T09:22:46.431300+0000 mon.a (mon.0) 546 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[35780]: cluster 2024-02-10T09:22:46.431385+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[35780]: audit 2024-02-10T09:22:46.433398+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[35780]: audit 2024-02-10T09:22:46.433821+0000 mon.a (mon.0) 548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[39117]: audit 2024-02-10T09:22:46.431300+0000 mon.a (mon.0) 546 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[39117]: cluster 2024-02-10T09:22:46.431385+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-10T09:22:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[39117]: audit 2024-02-10T09:22:46.433398+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T09:22:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:47 smithi153 bash[39117]: audit 2024-02-10T09:22:46.433821+0000 mon.a (mon.0) 548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T09:22:47.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:47 smithi204 bash[35809]: audit 2024-02-10T09:22:46.431300+0000 mon.a (mon.0) 546 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T09:22:47.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:47 smithi204 bash[35809]: cluster 2024-02-10T09:22:46.431385+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-10T09:22:47.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:47 smithi204 bash[35809]: audit 2024-02-10T09:22:46.433398+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T09:22:47.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:47 smithi204 bash[35809]: audit 2024-02-10T09:22:46.433821+0000 mon.a (mon.0) 548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "5c26173a-e0e7-4d4a-b925-93ec802dfe81", 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "predecessor_uuid": "533d482e-e080-4bd3-9fb0-5cc676d83a48", 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_status": [], 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "period_map": { 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "5c26173a-e0e7-4d4a-b925-93ec802dfe81", 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: "zonegroups": [ 2024-02-10T09:22:48.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "b5782749-01e9-4840-af19-5fe16b02c924", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "default", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "api_name": "default", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "is_master": "true", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "endpoints": [], 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "hostnames": [], 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "hostnames_s3website": [], 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "master_zone": "51ae5349-887c-4d78-95df-1d0f5a7e9eb5", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "zones": [ 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "id": "51ae5349-887c-4d78-95df-1d0f5a7e9eb5", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "z", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "endpoints": [], 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "log_meta": "false", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "log_data": "false", 2024-02-10T09:22:48.521 INFO:teuthology.orchestra.run.smithi153.stdout: "bucket_index_max_shards": 11, 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "read_only": "false", 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "tier_type": "", 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_from_all": "true", 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_from": [], 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "redirect_zone": "" 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "placement_targets": [ 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "default-placement", 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "tags": [], 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "storage_classes": [ 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: "STANDARD" 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-02-10T09:22:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "default_placement": "default-placement", 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "realm_id": "8a2fe70c-c3a7-4c18-b934-bbe49646e5fb", 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_policy": { 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "groups": [] 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "short_zone_ids": [ 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "key": "51ae5349-887c-4d78-95df-1d0f5a7e9eb5", 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 3551386310 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-02-10T09:22:48.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "master_zonegroup": "b5782749-01e9-4840-af19-5fe16b02c924", 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "master_zone": "51ae5349-887c-4d78-95df-1d0f5a7e9eb5", 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "period_config": { 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "bucket_quota": { 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "enabled": false, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "check_on_raw": false, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "max_size": -1, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "max_size_kb": 0, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "max_objects": -1 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "user_quota": { 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "enabled": false, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "check_on_raw": false, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "max_size": -1, 2024-02-10T09:22:48.524 INFO:teuthology.orchestra.run.smithi153.stdout: "max_size_kb": 0, 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout: "max_objects": -1 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout: "realm_id": "8a2fe70c-c3a7-4c18-b934-bbe49646e5fb", 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout: "realm_name": "r", 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout: "realm_epoch": 2 2024-02-10T09:22:48.525 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:22:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: audit 2024-02-10T09:22:47.435113+0000 mon.a (mon.0) 549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T09:22:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: cluster 2024-02-10T09:22:47.435175+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-10T09:22:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: audit 2024-02-10T09:22:47.436260+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T09:22:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: audit 2024-02-10T09:22:47.436746+0000 mon.a (mon.0) 551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T09:22:48.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: cluster 2024-02-10T09:22:47.797668+0000 mgr.y (mgr.14140) 206 : cluster [DBG] pgmap v215: 129 pgs: 8 creating+peering, 28 unknown, 93 active+clean; 1.8 KiB data, 4.6 MiB used, 707 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T09:22:48.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: audit 2024-02-10T09:22:48.433809+0000 mon.a (mon.0) 552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T09:22:48.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:48 smithi204 bash[35809]: cluster 2024-02-10T09:22:48.433868+0000 mon.a (mon.0) 553 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-10T09:22:48.859 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: audit 2024-02-10T09:22:47.435113+0000 mon.a (mon.0) 549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T09:22:48.859 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: cluster 2024-02-10T09:22:47.435175+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: audit 2024-02-10T09:22:47.436260+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: audit 2024-02-10T09:22:47.436746+0000 mon.a (mon.0) 551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: cluster 2024-02-10T09:22:47.797668+0000 mgr.y (mgr.14140) 206 : cluster [DBG] pgmap v215: 129 pgs: 8 creating+peering, 28 unknown, 93 active+clean; 1.8 KiB data, 4.6 MiB used, 707 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: audit 2024-02-10T09:22:48.433809+0000 mon.a (mon.0) 552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[35780]: cluster 2024-02-10T09:22:48.433868+0000 mon.a (mon.0) 553 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: audit 2024-02-10T09:22:47.435113+0000 mon.a (mon.0) 549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T09:22:48.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: cluster 2024-02-10T09:22:47.435175+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-10T09:22:48.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: audit 2024-02-10T09:22:47.436260+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.153:0/3656761991' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T09:22:48.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: audit 2024-02-10T09:22:47.436746+0000 mon.a (mon.0) 551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T09:22:48.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: cluster 2024-02-10T09:22:47.797668+0000 mgr.y (mgr.14140) 206 : cluster [DBG] pgmap v215: 129 pgs: 8 creating+peering, 28 unknown, 93 active+clean; 1.8 KiB data, 4.6 MiB used, 707 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T09:22:48.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: audit 2024-02-10T09:22:48.433809+0000 mon.a (mon.0) 552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T09:22:48.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:48 smithi153 bash[39117]: cluster 2024-02-10T09:22:48.433868+0000 mon.a (mon.0) 553 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-10T09:22:49.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch apply rgw r z --placement=2 --port=8000' 2024-02-10T09:22:50.132 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:49 smithi153 bash[35780]: cluster 2024-02-10T09:22:49.798033+0000 mgr.y (mgr.14140) 207 : cluster [DBG] pgmap v217: 129 pgs: 7 creating+peering, 3 unknown, 119 active+clean; 3.1 KiB data, 4.9 MiB used, 707 GiB / 715 GiB avail; 234 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T09:22:50.132 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:49 smithi153 bash[39117]: cluster 2024-02-10T09:22:49.798033+0000 mgr.y (mgr.14140) 207 : cluster [DBG] pgmap v217: 129 pgs: 7 creating+peering, 3 unknown, 119 active+clean; 3.1 KiB data, 4.9 MiB used, 707 GiB / 715 GiB avail; 234 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T09:22:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:49 smithi204 bash[35809]: cluster 2024-02-10T09:22:49.798033+0000 mgr.y (mgr.14140) 207 : cluster [DBG] pgmap v217: 129 pgs: 7 creating+peering, 3 unknown, 119 active+clean; 3.1 KiB data, 4.9 MiB used, 707 GiB / 715 GiB avail; 234 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T09:22:51.046 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled rgw update... 2024-02-10T09:22:51.388 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:51 smithi153 bash[39117]: audit 2024-02-10T09:22:51.041486+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.rgw.r.z","val":"{\"created\": \"2024-02-10T09:22:51.041162\", \"spec\": {\"placement\": {\"count\": 2}, \"rgw_frontend_port\": 8000, \"rgw_realm\": \"r\", \"rgw_zone\": \"z\", \"service_id\": \"r.z\", \"service_type\": \"rgw\"}}"}]: dispatch 2024-02-10T09:22:51.388 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:51 smithi153 bash[39117]: audit 2024-02-10T09:22:51.045075+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.rgw.r.z","val":"{\"created\": \"2024-02-10T09:22:51.041162\", \"spec\": {\"placement\": {\"count\": 2}, \"rgw_frontend_port\": 8000, \"rgw_realm\": \"r\", \"rgw_zone\": \"z\", \"service_id\": \"r.z\", \"service_type\": \"rgw\"}}"}]': finished 2024-02-10T09:22:51.389 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:51 smithi153 bash[35780]: audit 2024-02-10T09:22:51.041486+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.rgw.r.z","val":"{\"created\": \"2024-02-10T09:22:51.041162\", \"spec\": {\"placement\": {\"count\": 2}, \"rgw_frontend_port\": 8000, \"rgw_realm\": \"r\", \"rgw_zone\": \"z\", \"service_id\": \"r.z\", \"service_type\": \"rgw\"}}"}]: dispatch 2024-02-10T09:22:51.389 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:51 smithi153 bash[35780]: audit 2024-02-10T09:22:51.045075+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.rgw.r.z","val":"{\"created\": \"2024-02-10T09:22:51.041162\", \"spec\": {\"placement\": {\"count\": 2}, \"rgw_frontend_port\": 8000, \"rgw_realm\": \"r\", \"rgw_zone\": \"z\", \"service_id\": \"r.z\", \"service_type\": \"rgw\"}}"}]': finished 2024-02-10T09:22:51.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:51 smithi204 bash[35809]: audit 2024-02-10T09:22:51.041486+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/spec.rgw.r.z","val":"{\"created\": \"2024-02-10T09:22:51.041162\", \"spec\": {\"placement\": {\"count\": 2}, \"rgw_frontend_port\": 8000, \"rgw_realm\": \"r\", \"rgw_zone\": \"z\", \"service_id\": \"r.z\", \"service_type\": \"rgw\"}}"}]: dispatch 2024-02-10T09:22:51.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:51 smithi204 bash[35809]: audit 2024-02-10T09:22:51.045075+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/spec.rgw.r.z","val":"{\"created\": \"2024-02-10T09:22:51.041162\", \"spec\": {\"placement\": {\"count\": 2}, \"rgw_frontend_port\": 8000, \"rgw_realm\": \"r\", \"rgw_zone\": \"z\", \"service_id\": \"r.z\", \"service_type\": \"rgw\"}}"}]': finished 2024-02-10T09:22:51.614 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 180' 2024-02-10T09:22:52.885 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:52 smithi153 bash[35780]: audit 2024-02-10T09:22:51.040721+0000 mgr.y (mgr.14140) 208 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "realm_name": "r", "zone_name": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:52.885 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:52 smithi153 bash[35780]: cephadm 2024-02-10T09:22:51.041153+0000 mgr.y (mgr.14140) 209 : cephadm [INF] Saving service rgw.r.z spec with placement count:2 2024-02-10T09:22:52.885 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:52 smithi153 bash[35780]: cluster 2024-02-10T09:22:51.798416+0000 mgr.y (mgr.14140) 210 : cluster [DBG] pgmap v218: 129 pgs: 129 active+clean; 4.1 KiB data, 5.3 MiB used, 707 GiB / 715 GiB avail; 341 B/s rd, 3.0 KiB/s wr, 10 op/s 2024-02-10T09:22:52.885 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:52 smithi153 bash[39117]: audit 2024-02-10T09:22:51.040721+0000 mgr.y (mgr.14140) 208 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "realm_name": "r", "zone_name": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:52.885 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:52 smithi153 bash[39117]: cephadm 2024-02-10T09:22:51.041153+0000 mgr.y (mgr.14140) 209 : cephadm [INF] Saving service rgw.r.z spec with placement count:2 2024-02-10T09:22:52.885 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:52 smithi153 bash[39117]: cluster 2024-02-10T09:22:51.798416+0000 mgr.y (mgr.14140) 210 : cluster [DBG] pgmap v218: 129 pgs: 129 active+clean; 4.1 KiB data, 5.3 MiB used, 707 GiB / 715 GiB avail; 341 B/s rd, 3.0 KiB/s wr, 10 op/s 2024-02-10T09:22:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:52 smithi204 bash[35809]: audit 2024-02-10T09:22:51.040721+0000 mgr.y (mgr.14140) 208 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "realm_name": "r", "zone_name": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:52 smithi204 bash[35809]: cephadm 2024-02-10T09:22:51.041153+0000 mgr.y (mgr.14140) 209 : cephadm [INF] Saving service rgw.r.z spec with placement count:2 2024-02-10T09:22:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:52 smithi204 bash[35809]: cluster 2024-02-10T09:22:51.798416+0000 mgr.y (mgr.14140) 210 : cluster [DBG] pgmap v218: 129 pgs: 129 active+clean; 4.1 KiB data, 5.3 MiB used, 707 GiB / 715 GiB avail; 341 B/s rd, 3.0 KiB/s wr, 10 op/s 2024-02-10T09:22:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:53 smithi153 bash[35780]: cluster 2024-02-10T09:22:53.798912+0000 mgr.y (mgr.14140) 211 : cluster [DBG] pgmap v219: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 972 B/s rd, 5.3 KiB/s wr, 19 op/s 2024-02-10T09:22:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:53 smithi153 bash[39117]: cluster 2024-02-10T09:22:53.798912+0000 mgr.y (mgr.14140) 211 : cluster [DBG] pgmap v219: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 972 B/s rd, 5.3 KiB/s wr, 19 op/s 2024-02-10T09:22:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:53 smithi204 bash[35809]: cluster 2024-02-10T09:22:53.798912+0000 mgr.y (mgr.14140) 211 : cluster [DBG] pgmap v219: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 972 B/s rd, 5.3 KiB/s wr, 19 op/s 2024-02-10T09:22:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:55 smithi153 bash[35780]: cluster 2024-02-10T09:22:55.799396+0000 mgr.y (mgr.14140) 212 : cluster [DBG] pgmap v220: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 856 B/s rd, 4.7 KiB/s wr, 17 op/s 2024-02-10T09:22:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:55 smithi153 bash[39117]: cluster 2024-02-10T09:22:55.799396+0000 mgr.y (mgr.14140) 212 : cluster [DBG] pgmap v220: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 856 B/s rd, 4.7 KiB/s wr, 17 op/s 2024-02-10T09:22:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:55 smithi204 bash[35809]: cluster 2024-02-10T09:22:55.799396+0000 mgr.y (mgr.14140) 212 : cluster [DBG] pgmap v220: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 856 B/s rd, 4.7 KiB/s wr, 17 op/s 2024-02-10T09:22:58.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:57 smithi153 bash[35780]: cluster 2024-02-10T09:22:57.799868+0000 mgr.y (mgr.14140) 213 : cluster [DBG] pgmap v221: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 716 B/s rd, 3.9 KiB/s wr, 14 op/s 2024-02-10T09:22:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:57 smithi153 bash[39117]: cluster 2024-02-10T09:22:57.799868+0000 mgr.y (mgr.14140) 213 : cluster [DBG] pgmap v221: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 716 B/s rd, 3.9 KiB/s wr, 14 op/s 2024-02-10T09:22:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:57 smithi204 bash[35809]: cluster 2024-02-10T09:22:57.799868+0000 mgr.y (mgr.14140) 213 : cluster [DBG] pgmap v221: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 716 B/s rd, 3.9 KiB/s wr, 14 op/s 2024-02-10T09:23:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:22:59 smithi153 bash[35780]: cluster 2024-02-10T09:22:59.800587+0000 mgr.y (mgr.14140) 214 : cluster [DBG] pgmap v222: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 630 B/s rd, 3.4 KiB/s wr, 12 op/s 2024-02-10T09:23:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:22:59 smithi153 bash[39117]: cluster 2024-02-10T09:22:59.800587+0000 mgr.y (mgr.14140) 214 : cluster [DBG] pgmap v222: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 630 B/s rd, 3.4 KiB/s wr, 12 op/s 2024-02-10T09:23:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:22:59 smithi204 bash[35809]: cluster 2024-02-10T09:22:59.800587+0000 mgr.y (mgr.14140) 214 : cluster [DBG] pgmap v222: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 630 B/s rd, 3.4 KiB/s wr, 12 op/s 2024-02-10T09:23:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:02 smithi153 bash[35780]: cluster 2024-02-10T09:23:01.801120+0000 mgr.y (mgr.14140) 215 : cluster [DBG] pgmap v223: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 597 B/s rd, 2.8 KiB/s wr, 10 op/s 2024-02-10T09:23:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:02 smithi153 bash[39117]: cluster 2024-02-10T09:23:01.801120+0000 mgr.y (mgr.14140) 215 : cluster [DBG] pgmap v223: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 597 B/s rd, 2.8 KiB/s wr, 10 op/s 2024-02-10T09:23:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:02 smithi204 bash[35809]: cluster 2024-02-10T09:23:01.801120+0000 mgr.y (mgr.14140) 215 : cluster [DBG] pgmap v223: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 597 B/s rd, 2.8 KiB/s wr, 10 op/s 2024-02-10T09:23:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:03 smithi153 bash[35780]: cluster 2024-02-10T09:23:03.801979+0000 mgr.y (mgr.14140) 216 : cluster [DBG] pgmap v224: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 426 B/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-10T09:23:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:03 smithi153 bash[39117]: cluster 2024-02-10T09:23:03.801979+0000 mgr.y (mgr.14140) 216 : cluster [DBG] pgmap v224: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 426 B/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-10T09:23:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:03 smithi204 bash[35809]: cluster 2024-02-10T09:23:03.801979+0000 mgr.y (mgr.14140) 216 : cluster [DBG] pgmap v224: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 426 B/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-10T09:23:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:05 smithi153 bash[35780]: cluster 2024-02-10T09:23:05.802663+0000 mgr.y (mgr.14140) 217 : cluster [DBG] pgmap v225: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:05 smithi153 bash[39117]: cluster 2024-02-10T09:23:05.802663+0000 mgr.y (mgr.14140) 217 : cluster [DBG] pgmap v225: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:05 smithi204 bash[35809]: cluster 2024-02-10T09:23:05.802663+0000 mgr.y (mgr.14140) 217 : cluster [DBG] pgmap v225: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:07 smithi153 bash[39117]: cluster 2024-02-10T09:23:07.803135+0000 mgr.y (mgr.14140) 218 : cluster [DBG] pgmap v226: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:07 smithi153 bash[35780]: cluster 2024-02-10T09:23:07.803135+0000 mgr.y (mgr.14140) 218 : cluster [DBG] pgmap v226: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:07 smithi204 bash[35809]: cluster 2024-02-10T09:23:07.803135+0000 mgr.y (mgr.14140) 218 : cluster [DBG] pgmap v226: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:10.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:09 smithi153 bash[35780]: cluster 2024-02-10T09:23:09.803669+0000 mgr.y (mgr.14140) 219 : cluster [DBG] pgmap v227: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:09 smithi153 bash[39117]: cluster 2024-02-10T09:23:09.803669+0000 mgr.y (mgr.14140) 219 : cluster [DBG] pgmap v227: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:09 smithi204 bash[35809]: cluster 2024-02-10T09:23:09.803669+0000 mgr.y (mgr.14140) 219 : cluster [DBG] pgmap v227: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:12.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:12 smithi153 bash[35780]: cluster 2024-02-10T09:23:11.804411+0000 mgr.y (mgr.14140) 220 : cluster [DBG] pgmap v228: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:12.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:12 smithi153 bash[39117]: cluster 2024-02-10T09:23:11.804411+0000 mgr.y (mgr.14140) 220 : cluster [DBG] pgmap v228: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:13.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:12 smithi204 bash[35809]: cluster 2024-02-10T09:23:11.804411+0000 mgr.y (mgr.14140) 220 : cluster [DBG] pgmap v228: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:13 smithi153 bash[35780]: cluster 2024-02-10T09:23:13.805000+0000 mgr.y (mgr.14140) 221 : cluster [DBG] pgmap v229: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:13 smithi153 bash[39117]: cluster 2024-02-10T09:23:13.805000+0000 mgr.y (mgr.14140) 221 : cluster [DBG] pgmap v229: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:13 smithi204 bash[35809]: cluster 2024-02-10T09:23:13.805000+0000 mgr.y (mgr.14140) 221 : cluster [DBG] pgmap v229: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:15 smithi153 bash[35780]: cluster 2024-02-10T09:23:15.805574+0000 mgr.y (mgr.14140) 222 : cluster [DBG] pgmap v230: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:15 smithi153 bash[39117]: cluster 2024-02-10T09:23:15.805574+0000 mgr.y (mgr.14140) 222 : cluster [DBG] pgmap v230: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:15 smithi204 bash[35809]: cluster 2024-02-10T09:23:15.805574+0000 mgr.y (mgr.14140) 222 : cluster [DBG] pgmap v230: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:17 smithi153 bash[35780]: cluster 2024-02-10T09:23:17.806084+0000 mgr.y (mgr.14140) 223 : cluster [DBG] pgmap v231: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:17 smithi153 bash[39117]: cluster 2024-02-10T09:23:17.806084+0000 mgr.y (mgr.14140) 223 : cluster [DBG] pgmap v231: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:17 smithi204 bash[35809]: cluster 2024-02-10T09:23:17.806084+0000 mgr.y (mgr.14140) 223 : cluster [DBG] pgmap v231: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:19 smithi153 bash[35780]: cluster 2024-02-10T09:23:19.806889+0000 mgr.y (mgr.14140) 224 : cluster [DBG] pgmap v232: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:19 smithi153 bash[39117]: cluster 2024-02-10T09:23:19.806889+0000 mgr.y (mgr.14140) 224 : cluster [DBG] pgmap v232: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:19 smithi204 bash[35809]: cluster 2024-02-10T09:23:19.806889+0000 mgr.y (mgr.14140) 224 : cluster [DBG] pgmap v232: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:22 smithi153 bash[35780]: cluster 2024-02-10T09:23:21.807470+0000 mgr.y (mgr.14140) 225 : cluster [DBG] pgmap v233: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:22 smithi153 bash[39117]: cluster 2024-02-10T09:23:21.807470+0000 mgr.y (mgr.14140) 225 : cluster [DBG] pgmap v233: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:22 smithi204 bash[35809]: cluster 2024-02-10T09:23:21.807470+0000 mgr.y (mgr.14140) 225 : cluster [DBG] pgmap v233: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:24.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:23 smithi153 bash[35780]: cluster 2024-02-10T09:23:23.808124+0000 mgr.y (mgr.14140) 226 : cluster [DBG] pgmap v234: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:23 smithi153 bash[39117]: cluster 2024-02-10T09:23:23.808124+0000 mgr.y (mgr.14140) 226 : cluster [DBG] pgmap v234: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:23 smithi204 bash[35809]: cluster 2024-02-10T09:23:23.808124+0000 mgr.y (mgr.14140) 226 : cluster [DBG] pgmap v234: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:26.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:25 smithi153 bash[35780]: cluster 2024-02-10T09:23:25.808806+0000 mgr.y (mgr.14140) 227 : cluster [DBG] pgmap v235: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:26.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:25 smithi153 bash[39117]: cluster 2024-02-10T09:23:25.808806+0000 mgr.y (mgr.14140) 227 : cluster [DBG] pgmap v235: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:25 smithi204 bash[35809]: cluster 2024-02-10T09:23:25.808806+0000 mgr.y (mgr.14140) 227 : cluster [DBG] pgmap v235: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:27 smithi153 bash[35780]: cluster 2024-02-10T09:23:27.809485+0000 mgr.y (mgr.14140) 228 : cluster [DBG] pgmap v236: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:27 smithi153 bash[39117]: cluster 2024-02-10T09:23:27.809485+0000 mgr.y (mgr.14140) 228 : cluster [DBG] pgmap v236: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:27 smithi204 bash[35809]: cluster 2024-02-10T09:23:27.809485+0000 mgr.y (mgr.14140) 228 : cluster [DBG] pgmap v236: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:29 smithi153 bash[35780]: cluster 2024-02-10T09:23:29.810120+0000 mgr.y (mgr.14140) 229 : cluster [DBG] pgmap v237: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:29 smithi153 bash[39117]: cluster 2024-02-10T09:23:29.810120+0000 mgr.y (mgr.14140) 229 : cluster [DBG] pgmap v237: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:29 smithi204 bash[35809]: cluster 2024-02-10T09:23:29.810120+0000 mgr.y (mgr.14140) 229 : cluster [DBG] pgmap v237: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:32 smithi153 bash[35780]: cluster 2024-02-10T09:23:31.810688+0000 mgr.y (mgr.14140) 230 : cluster [DBG] pgmap v238: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:32 smithi153 bash[39117]: cluster 2024-02-10T09:23:31.810688+0000 mgr.y (mgr.14140) 230 : cluster [DBG] pgmap v238: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:32 smithi204 bash[35809]: cluster 2024-02-10T09:23:31.810688+0000 mgr.y (mgr.14140) 230 : cluster [DBG] pgmap v238: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:33 smithi153 bash[35780]: cluster 2024-02-10T09:23:33.811439+0000 mgr.y (mgr.14140) 231 : cluster [DBG] pgmap v239: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:33 smithi153 bash[39117]: cluster 2024-02-10T09:23:33.811439+0000 mgr.y (mgr.14140) 231 : cluster [DBG] pgmap v239: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:33 smithi204 bash[35809]: cluster 2024-02-10T09:23:33.811439+0000 mgr.y (mgr.14140) 231 : cluster [DBG] pgmap v239: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:35 smithi153 bash[35780]: cluster 2024-02-10T09:23:35.812135+0000 mgr.y (mgr.14140) 232 : cluster [DBG] pgmap v240: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:35 smithi153 bash[39117]: cluster 2024-02-10T09:23:35.812135+0000 mgr.y (mgr.14140) 232 : cluster [DBG] pgmap v240: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:35 smithi204 bash[35809]: cluster 2024-02-10T09:23:35.812135+0000 mgr.y (mgr.14140) 232 : cluster [DBG] pgmap v240: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:37 smithi153 bash[35780]: cluster 2024-02-10T09:23:37.812661+0000 mgr.y (mgr.14140) 233 : cluster [DBG] pgmap v241: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:37 smithi153 bash[39117]: cluster 2024-02-10T09:23:37.812661+0000 mgr.y (mgr.14140) 233 : cluster [DBG] pgmap v241: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:37 smithi204 bash[35809]: cluster 2024-02-10T09:23:37.812661+0000 mgr.y (mgr.14140) 233 : cluster [DBG] pgmap v241: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:39 smithi153 bash[35780]: cluster 2024-02-10T09:23:39.813383+0000 mgr.y (mgr.14140) 234 : cluster [DBG] pgmap v242: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:39 smithi153 bash[39117]: cluster 2024-02-10T09:23:39.813383+0000 mgr.y (mgr.14140) 234 : cluster [DBG] pgmap v242: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:39 smithi204 bash[35809]: cluster 2024-02-10T09:23:39.813383+0000 mgr.y (mgr.14140) 234 : cluster [DBG] pgmap v242: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:42.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:42 smithi153 bash[35780]: cluster 2024-02-10T09:23:41.814029+0000 mgr.y (mgr.14140) 235 : cluster [DBG] pgmap v243: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:42.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:42 smithi153 bash[39117]: cluster 2024-02-10T09:23:41.814029+0000 mgr.y (mgr.14140) 235 : cluster [DBG] pgmap v243: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:42 smithi204 bash[35809]: cluster 2024-02-10T09:23:41.814029+0000 mgr.y (mgr.14140) 235 : cluster [DBG] pgmap v243: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:43 smithi153 bash[35780]: cluster 2024-02-10T09:23:43.814644+0000 mgr.y (mgr.14140) 236 : cluster [DBG] pgmap v244: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:43 smithi153 bash[39117]: cluster 2024-02-10T09:23:43.814644+0000 mgr.y (mgr.14140) 236 : cluster [DBG] pgmap v244: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:43 smithi204 bash[35809]: cluster 2024-02-10T09:23:43.814644+0000 mgr.y (mgr.14140) 236 : cluster [DBG] pgmap v244: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: debug 2024-02-10T09:23:44.219+0000 7f90e3a5e700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:44.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 4247, in 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 968, in _default_image 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 2508, in command_deploy 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 1974, in run 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 696, in call_throws 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: code, '\n'.join(err))) 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:44.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:23:44.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 4247, in 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 968, in _default_image 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 2508, in command_deploy 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 1974, in run 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: File "", line 696, in call_throws 2024-02-10T09:23:44.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[36014]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: cephadm 2024-02-10T09:23:44.221628+0000 mgr.y (mgr.14140) 237 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: cephadm 2024-02-10T09:23:44.221628+0000 mgr.y (mgr.14140) 237 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: cephadm 2024-02-10T09:23:44.221822+0000 mgr.y (mgr.14140) 238 : cephadm [WRN] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi204', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: cephadm 2024-02-10T09:23:44.221822+0000 mgr.y (mgr.14140) 238 : cephadm [WRN] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi204', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[39117]: cephadm 2024-02-10T09:23:44.225219+0000 mgr.y (mgr.14140) 239 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:23:45.200 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.201 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.201 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.201 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:23:45.201 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:23:45.201 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.201 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:44 smithi153 bash[35780]: cephadm 2024-02-10T09:23:44.225219+0000 mgr.y (mgr.14140) 239 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: cephadm 2024-02-10T09:23:44.221628+0000 mgr.y (mgr.14140) 237 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:23:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: cephadm 2024-02-10T09:23:44.221822+0000 mgr.y (mgr.14140) 238 : cephadm [WRN] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi204', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:21:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:22:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:23:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:23:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:23:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:23:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:23:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:44 smithi204 bash[35809]: cephadm 2024-02-10T09:23:44.225219+0000 mgr.y (mgr.14140) 239 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:23:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:45 smithi153 bash[35780]: cluster 2024-02-10T09:23:45.815218+0000 mgr.y (mgr.14140) 240 : cluster [DBG] pgmap v245: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:45 smithi153 bash[39117]: cluster 2024-02-10T09:23:45.815218+0000 mgr.y (mgr.14140) 240 : cluster [DBG] pgmap v245: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:45 smithi204 bash[35809]: cluster 2024-02-10T09:23:45.815218+0000 mgr.y (mgr.14140) 240 : cluster [DBG] pgmap v245: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:47 smithi153 bash[35780]: cluster 2024-02-10T09:23:47.815924+0000 mgr.y (mgr.14140) 241 : cluster [DBG] pgmap v246: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:47 smithi153 bash[39117]: cluster 2024-02-10T09:23:47.815924+0000 mgr.y (mgr.14140) 241 : cluster [DBG] pgmap v246: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:47 smithi204 bash[35809]: cluster 2024-02-10T09:23:47.815924+0000 mgr.y (mgr.14140) 241 : cluster [DBG] pgmap v246: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:49 smithi153 bash[35780]: cluster 2024-02-10T09:23:49.816528+0000 mgr.y (mgr.14140) 242 : cluster [DBG] pgmap v247: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:49 smithi153 bash[39117]: cluster 2024-02-10T09:23:49.816528+0000 mgr.y (mgr.14140) 242 : cluster [DBG] pgmap v247: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:49 smithi204 bash[35809]: cluster 2024-02-10T09:23:49.816528+0000 mgr.y (mgr.14140) 242 : cluster [DBG] pgmap v247: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:52 smithi153 bash[35780]: cluster 2024-02-10T09:23:51.817133+0000 mgr.y (mgr.14140) 243 : cluster [DBG] pgmap v248: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:52 smithi153 bash[39117]: cluster 2024-02-10T09:23:51.817133+0000 mgr.y (mgr.14140) 243 : cluster [DBG] pgmap v248: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:52 smithi204 bash[35809]: cluster 2024-02-10T09:23:51.817133+0000 mgr.y (mgr.14140) 243 : cluster [DBG] pgmap v248: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:53 smithi153 bash[35780]: cluster 2024-02-10T09:23:53.817674+0000 mgr.y (mgr.14140) 244 : cluster [DBG] pgmap v249: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:53 smithi153 bash[39117]: cluster 2024-02-10T09:23:53.817674+0000 mgr.y (mgr.14140) 244 : cluster [DBG] pgmap v249: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:53 smithi204 bash[35809]: cluster 2024-02-10T09:23:53.817674+0000 mgr.y (mgr.14140) 244 : cluster [DBG] pgmap v249: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:56.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:55 smithi153 bash[35780]: cluster 2024-02-10T09:23:55.818190+0000 mgr.y (mgr.14140) 245 : cluster [DBG] pgmap v250: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:55 smithi153 bash[39117]: cluster 2024-02-10T09:23:55.818190+0000 mgr.y (mgr.14140) 245 : cluster [DBG] pgmap v250: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:55 smithi204 bash[35809]: cluster 2024-02-10T09:23:55.818190+0000 mgr.y (mgr.14140) 245 : cluster [DBG] pgmap v250: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:57 smithi153 bash[35780]: cluster 2024-02-10T09:23:57.818863+0000 mgr.y (mgr.14140) 246 : cluster [DBG] pgmap v251: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:57 smithi153 bash[39117]: cluster 2024-02-10T09:23:57.818863+0000 mgr.y (mgr.14140) 246 : cluster [DBG] pgmap v251: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:23:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:57 smithi204 bash[35809]: cluster 2024-02-10T09:23:57.818863+0000 mgr.y (mgr.14140) 246 : cluster [DBG] pgmap v251: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:23:59 smithi153 bash[39117]: cluster 2024-02-10T09:23:59.819459+0000 mgr.y (mgr.14140) 247 : cluster [DBG] pgmap v252: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:23:59 smithi153 bash[35780]: cluster 2024-02-10T09:23:59.819459+0000 mgr.y (mgr.14140) 247 : cluster [DBG] pgmap v252: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:23:59 smithi204 bash[35809]: cluster 2024-02-10T09:23:59.819459+0000 mgr.y (mgr.14140) 247 : cluster [DBG] pgmap v252: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:02 smithi153 bash[35780]: cluster 2024-02-10T09:24:01.820113+0000 mgr.y (mgr.14140) 248 : cluster [DBG] pgmap v253: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:02 smithi153 bash[39117]: cluster 2024-02-10T09:24:01.820113+0000 mgr.y (mgr.14140) 248 : cluster [DBG] pgmap v253: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:02 smithi204 bash[35809]: cluster 2024-02-10T09:24:01.820113+0000 mgr.y (mgr.14140) 248 : cluster [DBG] pgmap v253: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:04.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:03 smithi153 bash[35780]: cluster 2024-02-10T09:24:03.820873+0000 mgr.y (mgr.14140) 249 : cluster [DBG] pgmap v254: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:03 smithi153 bash[39117]: cluster 2024-02-10T09:24:03.820873+0000 mgr.y (mgr.14140) 249 : cluster [DBG] pgmap v254: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:03 smithi204 bash[35809]: cluster 2024-02-10T09:24:03.820873+0000 mgr.y (mgr.14140) 249 : cluster [DBG] pgmap v254: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:05 smithi153 bash[35780]: cluster 2024-02-10T09:24:05.821410+0000 mgr.y (mgr.14140) 250 : cluster [DBG] pgmap v255: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:05 smithi153 bash[39117]: cluster 2024-02-10T09:24:05.821410+0000 mgr.y (mgr.14140) 250 : cluster [DBG] pgmap v255: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:05 smithi204 bash[35809]: cluster 2024-02-10T09:24:05.821410+0000 mgr.y (mgr.14140) 250 : cluster [DBG] pgmap v255: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:07 smithi153 bash[35780]: cluster 2024-02-10T09:24:07.822135+0000 mgr.y (mgr.14140) 251 : cluster [DBG] pgmap v256: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:07 smithi153 bash[39117]: cluster 2024-02-10T09:24:07.822135+0000 mgr.y (mgr.14140) 251 : cluster [DBG] pgmap v256: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:07 smithi204 bash[35809]: cluster 2024-02-10T09:24:07.822135+0000 mgr.y (mgr.14140) 251 : cluster [DBG] pgmap v256: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:10.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:09 smithi153 bash[39117]: cluster 2024-02-10T09:24:09.822776+0000 mgr.y (mgr.14140) 252 : cluster [DBG] pgmap v257: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:09 smithi153 bash[35780]: cluster 2024-02-10T09:24:09.822776+0000 mgr.y (mgr.14140) 252 : cluster [DBG] pgmap v257: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:09 smithi204 bash[35809]: cluster 2024-02-10T09:24:09.822776+0000 mgr.y (mgr.14140) 252 : cluster [DBG] pgmap v257: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:12.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:12 smithi153 bash[35780]: cluster 2024-02-10T09:24:11.823470+0000 mgr.y (mgr.14140) 253 : cluster [DBG] pgmap v258: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:12.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:12 smithi153 bash[39117]: cluster 2024-02-10T09:24:11.823470+0000 mgr.y (mgr.14140) 253 : cluster [DBG] pgmap v258: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:12 smithi204 bash[35809]: cluster 2024-02-10T09:24:11.823470+0000 mgr.y (mgr.14140) 253 : cluster [DBG] pgmap v258: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:13 smithi153 bash[35780]: cluster 2024-02-10T09:24:13.824116+0000 mgr.y (mgr.14140) 254 : cluster [DBG] pgmap v259: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:13 smithi153 bash[39117]: cluster 2024-02-10T09:24:13.824116+0000 mgr.y (mgr.14140) 254 : cluster [DBG] pgmap v259: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:13 smithi204 bash[35809]: cluster 2024-02-10T09:24:13.824116+0000 mgr.y (mgr.14140) 254 : cluster [DBG] pgmap v259: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:16.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:15 smithi153 bash[35780]: cluster 2024-02-10T09:24:15.824806+0000 mgr.y (mgr.14140) 255 : cluster [DBG] pgmap v260: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:15 smithi153 bash[39117]: cluster 2024-02-10T09:24:15.824806+0000 mgr.y (mgr.14140) 255 : cluster [DBG] pgmap v260: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:15 smithi204 bash[35809]: cluster 2024-02-10T09:24:15.824806+0000 mgr.y (mgr.14140) 255 : cluster [DBG] pgmap v260: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:17 smithi153 bash[35780]: cluster 2024-02-10T09:24:17.825275+0000 mgr.y (mgr.14140) 256 : cluster [DBG] pgmap v261: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:17 smithi153 bash[39117]: cluster 2024-02-10T09:24:17.825275+0000 mgr.y (mgr.14140) 256 : cluster [DBG] pgmap v261: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:17 smithi204 bash[35809]: cluster 2024-02-10T09:24:17.825275+0000 mgr.y (mgr.14140) 256 : cluster [DBG] pgmap v261: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:19 smithi153 bash[35780]: cluster 2024-02-10T09:24:19.825941+0000 mgr.y (mgr.14140) 257 : cluster [DBG] pgmap v262: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:19 smithi153 bash[39117]: cluster 2024-02-10T09:24:19.825941+0000 mgr.y (mgr.14140) 257 : cluster [DBG] pgmap v262: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:19 smithi204 bash[35809]: cluster 2024-02-10T09:24:19.825941+0000 mgr.y (mgr.14140) 257 : cluster [DBG] pgmap v262: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:22 smithi153 bash[35780]: cluster 2024-02-10T09:24:21.826618+0000 mgr.y (mgr.14140) 258 : cluster [DBG] pgmap v263: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:22 smithi153 bash[39117]: cluster 2024-02-10T09:24:21.826618+0000 mgr.y (mgr.14140) 258 : cluster [DBG] pgmap v263: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:22 smithi204 bash[35809]: cluster 2024-02-10T09:24:21.826618+0000 mgr.y (mgr.14140) 258 : cluster [DBG] pgmap v263: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:23 smithi153 bash[35780]: cluster 2024-02-10T09:24:23.827144+0000 mgr.y (mgr.14140) 259 : cluster [DBG] pgmap v264: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:23 smithi153 bash[39117]: cluster 2024-02-10T09:24:23.827144+0000 mgr.y (mgr.14140) 259 : cluster [DBG] pgmap v264: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:23 smithi204 bash[35809]: cluster 2024-02-10T09:24:23.827144+0000 mgr.y (mgr.14140) 259 : cluster [DBG] pgmap v264: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:25 smithi153 bash[35780]: cluster 2024-02-10T09:24:25.827914+0000 mgr.y (mgr.14140) 260 : cluster [DBG] pgmap v265: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:25 smithi153 bash[39117]: cluster 2024-02-10T09:24:25.827914+0000 mgr.y (mgr.14140) 260 : cluster [DBG] pgmap v265: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:25 smithi204 bash[35809]: cluster 2024-02-10T09:24:25.827914+0000 mgr.y (mgr.14140) 260 : cluster [DBG] pgmap v265: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:27 smithi153 bash[35780]: cluster 2024-02-10T09:24:27.828623+0000 mgr.y (mgr.14140) 261 : cluster [DBG] pgmap v266: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:27 smithi153 bash[39117]: cluster 2024-02-10T09:24:27.828623+0000 mgr.y (mgr.14140) 261 : cluster [DBG] pgmap v266: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:27 smithi204 bash[35809]: cluster 2024-02-10T09:24:27.828623+0000 mgr.y (mgr.14140) 261 : cluster [DBG] pgmap v266: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:29 smithi153 bash[35780]: cluster 2024-02-10T09:24:29.829251+0000 mgr.y (mgr.14140) 262 : cluster [DBG] pgmap v267: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:29 smithi153 bash[39117]: cluster 2024-02-10T09:24:29.829251+0000 mgr.y (mgr.14140) 262 : cluster [DBG] pgmap v267: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:29 smithi204 bash[35809]: cluster 2024-02-10T09:24:29.829251+0000 mgr.y (mgr.14140) 262 : cluster [DBG] pgmap v267: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:32 smithi153 bash[35780]: cluster 2024-02-10T09:24:31.829960+0000 mgr.y (mgr.14140) 263 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:32 smithi153 bash[39117]: cluster 2024-02-10T09:24:31.829960+0000 mgr.y (mgr.14140) 263 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:32 smithi204 bash[35809]: cluster 2024-02-10T09:24:31.829960+0000 mgr.y (mgr.14140) 263 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:34.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:33 smithi153 bash[35780]: cluster 2024-02-10T09:24:33.830692+0000 mgr.y (mgr.14140) 264 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:33 smithi153 bash[39117]: cluster 2024-02-10T09:24:33.830692+0000 mgr.y (mgr.14140) 264 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:33 smithi204 bash[35809]: cluster 2024-02-10T09:24:33.830692+0000 mgr.y (mgr.14140) 264 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:35 smithi153 bash[35780]: cluster 2024-02-10T09:24:35.831288+0000 mgr.y (mgr.14140) 265 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:35 smithi153 bash[39117]: cluster 2024-02-10T09:24:35.831288+0000 mgr.y (mgr.14140) 265 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:35 smithi204 bash[35809]: cluster 2024-02-10T09:24:35.831288+0000 mgr.y (mgr.14140) 265 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:37 smithi153 bash[35780]: cluster 2024-02-10T09:24:37.831768+0000 mgr.y (mgr.14140) 266 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:38.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:37 smithi153 bash[39117]: cluster 2024-02-10T09:24:37.831768+0000 mgr.y (mgr.14140) 266 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:37 smithi204 bash[35809]: cluster 2024-02-10T09:24:37.831768+0000 mgr.y (mgr.14140) 266 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:39 smithi153 bash[35780]: cluster 2024-02-10T09:24:39.832395+0000 mgr.y (mgr.14140) 267 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:39 smithi153 bash[39117]: cluster 2024-02-10T09:24:39.832395+0000 mgr.y (mgr.14140) 267 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:39 smithi204 bash[35809]: cluster 2024-02-10T09:24:39.832395+0000 mgr.y (mgr.14140) 267 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:42.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:42 smithi153 bash[39117]: cluster 2024-02-10T09:24:41.832971+0000 mgr.y (mgr.14140) 268 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:42.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:42 smithi153 bash[35780]: cluster 2024-02-10T09:24:41.832971+0000 mgr.y (mgr.14140) 268 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:42 smithi204 bash[35809]: cluster 2024-02-10T09:24:41.832971+0000 mgr.y (mgr.14140) 268 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:43 smithi153 bash[35780]: cluster 2024-02-10T09:24:43.833433+0000 mgr.y (mgr.14140) 269 : cluster [DBG] pgmap v274: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:43 smithi153 bash[39117]: cluster 2024-02-10T09:24:43.833433+0000 mgr.y (mgr.14140) 269 : cluster [DBG] pgmap v274: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:43 smithi204 bash[35809]: cluster 2024-02-10T09:24:43.833433+0000 mgr.y (mgr.14140) 269 : cluster [DBG] pgmap v274: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:45 smithi153 bash[35780]: cluster 2024-02-10T09:24:45.834092+0000 mgr.y (mgr.14140) 270 : cluster [DBG] pgmap v275: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:45 smithi153 bash[39117]: cluster 2024-02-10T09:24:45.834092+0000 mgr.y (mgr.14140) 270 : cluster [DBG] pgmap v275: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:45 smithi204 bash[35809]: cluster 2024-02-10T09:24:45.834092+0000 mgr.y (mgr.14140) 270 : cluster [DBG] pgmap v275: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:47 smithi153 bash[35780]: cluster 2024-02-10T09:24:47.834771+0000 mgr.y (mgr.14140) 271 : cluster [DBG] pgmap v276: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:47 smithi153 bash[39117]: cluster 2024-02-10T09:24:47.834771+0000 mgr.y (mgr.14140) 271 : cluster [DBG] pgmap v276: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:47 smithi204 bash[35809]: cluster 2024-02-10T09:24:47.834771+0000 mgr.y (mgr.14140) 271 : cluster [DBG] pgmap v276: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:49 smithi153 bash[35780]: cluster 2024-02-10T09:24:49.835259+0000 mgr.y (mgr.14140) 272 : cluster [DBG] pgmap v277: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:49 smithi153 bash[39117]: cluster 2024-02-10T09:24:49.835259+0000 mgr.y (mgr.14140) 272 : cluster [DBG] pgmap v277: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:49 smithi204 bash[35809]: cluster 2024-02-10T09:24:49.835259+0000 mgr.y (mgr.14140) 272 : cluster [DBG] pgmap v277: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:52 smithi153 bash[35780]: cluster 2024-02-10T09:24:51.835966+0000 mgr.y (mgr.14140) 273 : cluster [DBG] pgmap v278: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:52 smithi153 bash[39117]: cluster 2024-02-10T09:24:51.835966+0000 mgr.y (mgr.14140) 273 : cluster [DBG] pgmap v278: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:52 smithi204 bash[35809]: cluster 2024-02-10T09:24:51.835966+0000 mgr.y (mgr.14140) 273 : cluster [DBG] pgmap v278: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:53 smithi153 bash[35780]: cluster 2024-02-10T09:24:53.836514+0000 mgr.y (mgr.14140) 274 : cluster [DBG] pgmap v279: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:53 smithi153 bash[39117]: cluster 2024-02-10T09:24:53.836514+0000 mgr.y (mgr.14140) 274 : cluster [DBG] pgmap v279: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:54.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:53 smithi204 bash[35809]: cluster 2024-02-10T09:24:53.836514+0000 mgr.y (mgr.14140) 274 : cluster [DBG] pgmap v279: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:55 smithi153 bash[35780]: cluster 2024-02-10T09:24:55.837078+0000 mgr.y (mgr.14140) 275 : cluster [DBG] pgmap v280: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:55 smithi153 bash[39117]: cluster 2024-02-10T09:24:55.837078+0000 mgr.y (mgr.14140) 275 : cluster [DBG] pgmap v280: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:55 smithi204 bash[35809]: cluster 2024-02-10T09:24:55.837078+0000 mgr.y (mgr.14140) 275 : cluster [DBG] pgmap v280: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:57 smithi153 bash[35780]: cluster 2024-02-10T09:24:57.837697+0000 mgr.y (mgr.14140) 276 : cluster [DBG] pgmap v281: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:57 smithi153 bash[39117]: cluster 2024-02-10T09:24:57.837697+0000 mgr.y (mgr.14140) 276 : cluster [DBG] pgmap v281: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:24:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:57 smithi204 bash[35809]: cluster 2024-02-10T09:24:57.837697+0000 mgr.y (mgr.14140) 276 : cluster [DBG] pgmap v281: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:24:59 smithi153 bash[35780]: cluster 2024-02-10T09:24:59.838381+0000 mgr.y (mgr.14140) 277 : cluster [DBG] pgmap v282: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:24:59 smithi153 bash[39117]: cluster 2024-02-10T09:24:59.838381+0000 mgr.y (mgr.14140) 277 : cluster [DBG] pgmap v282: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:24:59 smithi204 bash[35809]: cluster 2024-02-10T09:24:59.838381+0000 mgr.y (mgr.14140) 277 : cluster [DBG] pgmap v282: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:02 smithi153 bash[35780]: cluster 2024-02-10T09:25:01.839005+0000 mgr.y (mgr.14140) 278 : cluster [DBG] pgmap v283: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:02 smithi153 bash[39117]: cluster 2024-02-10T09:25:01.839005+0000 mgr.y (mgr.14140) 278 : cluster [DBG] pgmap v283: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:02 smithi204 bash[35809]: cluster 2024-02-10T09:25:01.839005+0000 mgr.y (mgr.14140) 278 : cluster [DBG] pgmap v283: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:03 smithi153 bash[35780]: cluster 2024-02-10T09:25:03.839767+0000 mgr.y (mgr.14140) 279 : cluster [DBG] pgmap v284: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:03 smithi153 bash[39117]: cluster 2024-02-10T09:25:03.839767+0000 mgr.y (mgr.14140) 279 : cluster [DBG] pgmap v284: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:03 smithi204 bash[35809]: cluster 2024-02-10T09:25:03.839767+0000 mgr.y (mgr.14140) 279 : cluster [DBG] pgmap v284: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:05 smithi153 bash[35780]: cluster 2024-02-10T09:25:05.840445+0000 mgr.y (mgr.14140) 280 : cluster [DBG] pgmap v285: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:05 smithi153 bash[39117]: cluster 2024-02-10T09:25:05.840445+0000 mgr.y (mgr.14140) 280 : cluster [DBG] pgmap v285: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:05 smithi204 bash[35809]: cluster 2024-02-10T09:25:05.840445+0000 mgr.y (mgr.14140) 280 : cluster [DBG] pgmap v285: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:07 smithi153 bash[35780]: cluster 2024-02-10T09:25:07.841058+0000 mgr.y (mgr.14140) 281 : cluster [DBG] pgmap v286: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:07 smithi153 bash[39117]: cluster 2024-02-10T09:25:07.841058+0000 mgr.y (mgr.14140) 281 : cluster [DBG] pgmap v286: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:07 smithi204 bash[35809]: cluster 2024-02-10T09:25:07.841058+0000 mgr.y (mgr.14140) 281 : cluster [DBG] pgmap v286: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:09 smithi153 bash[35780]: cluster 2024-02-10T09:25:09.841592+0000 mgr.y (mgr.14140) 282 : cluster [DBG] pgmap v287: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:09 smithi153 bash[39117]: cluster 2024-02-10T09:25:09.841592+0000 mgr.y (mgr.14140) 282 : cluster [DBG] pgmap v287: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:09 smithi204 bash[35809]: cluster 2024-02-10T09:25:09.841592+0000 mgr.y (mgr.14140) 282 : cluster [DBG] pgmap v287: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:12.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:12 smithi153 bash[35780]: cluster 2024-02-10T09:25:11.842360+0000 mgr.y (mgr.14140) 283 : cluster [DBG] pgmap v288: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:12.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:12 smithi153 bash[39117]: cluster 2024-02-10T09:25:11.842360+0000 mgr.y (mgr.14140) 283 : cluster [DBG] pgmap v288: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:12 smithi204 bash[35809]: cluster 2024-02-10T09:25:11.842360+0000 mgr.y (mgr.14140) 283 : cluster [DBG] pgmap v288: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:13 smithi153 bash[35780]: cluster 2024-02-10T09:25:13.843011+0000 mgr.y (mgr.14140) 284 : cluster [DBG] pgmap v289: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:13 smithi153 bash[39117]: cluster 2024-02-10T09:25:13.843011+0000 mgr.y (mgr.14140) 284 : cluster [DBG] pgmap v289: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:13 smithi204 bash[35809]: cluster 2024-02-10T09:25:13.843011+0000 mgr.y (mgr.14140) 284 : cluster [DBG] pgmap v289: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:15 smithi153 bash[35780]: cluster 2024-02-10T09:25:15.843668+0000 mgr.y (mgr.14140) 285 : cluster [DBG] pgmap v290: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:15 smithi153 bash[39117]: cluster 2024-02-10T09:25:15.843668+0000 mgr.y (mgr.14140) 285 : cluster [DBG] pgmap v290: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:15 smithi204 bash[35809]: cluster 2024-02-10T09:25:15.843668+0000 mgr.y (mgr.14140) 285 : cluster [DBG] pgmap v290: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:17 smithi153 bash[35780]: cluster 2024-02-10T09:25:17.844260+0000 mgr.y (mgr.14140) 286 : cluster [DBG] pgmap v291: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:17 smithi153 bash[39117]: cluster 2024-02-10T09:25:17.844260+0000 mgr.y (mgr.14140) 286 : cluster [DBG] pgmap v291: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:17 smithi204 bash[35809]: cluster 2024-02-10T09:25:17.844260+0000 mgr.y (mgr.14140) 286 : cluster [DBG] pgmap v291: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:19 smithi153 bash[35780]: cluster 2024-02-10T09:25:19.844719+0000 mgr.y (mgr.14140) 287 : cluster [DBG] pgmap v292: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:19 smithi153 bash[39117]: cluster 2024-02-10T09:25:19.844719+0000 mgr.y (mgr.14140) 287 : cluster [DBG] pgmap v292: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:19 smithi204 bash[35809]: cluster 2024-02-10T09:25:19.844719+0000 mgr.y (mgr.14140) 287 : cluster [DBG] pgmap v292: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:22 smithi153 bash[35780]: cluster 2024-02-10T09:25:21.845426+0000 mgr.y (mgr.14140) 288 : cluster [DBG] pgmap v293: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:22 smithi153 bash[39117]: cluster 2024-02-10T09:25:21.845426+0000 mgr.y (mgr.14140) 288 : cluster [DBG] pgmap v293: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:22 smithi204 bash[35809]: cluster 2024-02-10T09:25:21.845426+0000 mgr.y (mgr.14140) 288 : cluster [DBG] pgmap v293: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:24.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:23 smithi153 bash[39117]: cluster 2024-02-10T09:25:23.846089+0000 mgr.y (mgr.14140) 289 : cluster [DBG] pgmap v294: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:23 smithi153 bash[35780]: cluster 2024-02-10T09:25:23.846089+0000 mgr.y (mgr.14140) 289 : cluster [DBG] pgmap v294: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:23 smithi204 bash[35809]: cluster 2024-02-10T09:25:23.846089+0000 mgr.y (mgr.14140) 289 : cluster [DBG] pgmap v294: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:26.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:25 smithi153 bash[35780]: cluster 2024-02-10T09:25:25.846641+0000 mgr.y (mgr.14140) 290 : cluster [DBG] pgmap v295: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:25 smithi153 bash[39117]: cluster 2024-02-10T09:25:25.846641+0000 mgr.y (mgr.14140) 290 : cluster [DBG] pgmap v295: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:25 smithi204 bash[35809]: cluster 2024-02-10T09:25:25.846641+0000 mgr.y (mgr.14140) 290 : cluster [DBG] pgmap v295: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:28.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:27 smithi153 bash[35780]: cluster 2024-02-10T09:25:27.847141+0000 mgr.y (mgr.14140) 291 : cluster [DBG] pgmap v296: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:27 smithi153 bash[39117]: cluster 2024-02-10T09:25:27.847141+0000 mgr.y (mgr.14140) 291 : cluster [DBG] pgmap v296: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:27 smithi204 bash[35809]: cluster 2024-02-10T09:25:27.847141+0000 mgr.y (mgr.14140) 291 : cluster [DBG] pgmap v296: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:29 smithi153 bash[35780]: cluster 2024-02-10T09:25:29.847738+0000 mgr.y (mgr.14140) 292 : cluster [DBG] pgmap v297: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:29 smithi153 bash[39117]: cluster 2024-02-10T09:25:29.847738+0000 mgr.y (mgr.14140) 292 : cluster [DBG] pgmap v297: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:29 smithi204 bash[35809]: cluster 2024-02-10T09:25:29.847738+0000 mgr.y (mgr.14140) 292 : cluster [DBG] pgmap v297: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:32 smithi153 bash[35780]: cluster 2024-02-10T09:25:31.848276+0000 mgr.y (mgr.14140) 293 : cluster [DBG] pgmap v298: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:32 smithi153 bash[39117]: cluster 2024-02-10T09:25:31.848276+0000 mgr.y (mgr.14140) 293 : cluster [DBG] pgmap v298: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:32 smithi204 bash[35809]: cluster 2024-02-10T09:25:31.848276+0000 mgr.y (mgr.14140) 293 : cluster [DBG] pgmap v298: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:33 smithi153 bash[39117]: cluster 2024-02-10T09:25:33.848954+0000 mgr.y (mgr.14140) 294 : cluster [DBG] pgmap v299: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:33 smithi153 bash[35780]: cluster 2024-02-10T09:25:33.848954+0000 mgr.y (mgr.14140) 294 : cluster [DBG] pgmap v299: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:33 smithi204 bash[35809]: cluster 2024-02-10T09:25:33.848954+0000 mgr.y (mgr.14140) 294 : cluster [DBG] pgmap v299: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:35 smithi153 bash[35780]: cluster 2024-02-10T09:25:35.849600+0000 mgr.y (mgr.14140) 295 : cluster [DBG] pgmap v300: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:35 smithi153 bash[39117]: cluster 2024-02-10T09:25:35.849600+0000 mgr.y (mgr.14140) 295 : cluster [DBG] pgmap v300: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:35 smithi204 bash[35809]: cluster 2024-02-10T09:25:35.849600+0000 mgr.y (mgr.14140) 295 : cluster [DBG] pgmap v300: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:37 smithi153 bash[35780]: cluster 2024-02-10T09:25:37.850133+0000 mgr.y (mgr.14140) 296 : cluster [DBG] pgmap v301: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:37 smithi153 bash[39117]: cluster 2024-02-10T09:25:37.850133+0000 mgr.y (mgr.14140) 296 : cluster [DBG] pgmap v301: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:37 smithi204 bash[35809]: cluster 2024-02-10T09:25:37.850133+0000 mgr.y (mgr.14140) 296 : cluster [DBG] pgmap v301: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:39 smithi153 bash[35780]: cluster 2024-02-10T09:25:39.850735+0000 mgr.y (mgr.14140) 297 : cluster [DBG] pgmap v302: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:39 smithi153 bash[39117]: cluster 2024-02-10T09:25:39.850735+0000 mgr.y (mgr.14140) 297 : cluster [DBG] pgmap v302: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:39 smithi204 bash[35809]: cluster 2024-02-10T09:25:39.850735+0000 mgr.y (mgr.14140) 297 : cluster [DBG] pgmap v302: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:42.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:42 smithi153 bash[35780]: cluster 2024-02-10T09:25:41.851253+0000 mgr.y (mgr.14140) 298 : cluster [DBG] pgmap v303: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:42.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:42 smithi153 bash[39117]: cluster 2024-02-10T09:25:41.851253+0000 mgr.y (mgr.14140) 298 : cluster [DBG] pgmap v303: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:42 smithi204 bash[35809]: cluster 2024-02-10T09:25:41.851253+0000 mgr.y (mgr.14140) 298 : cluster [DBG] pgmap v303: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:43 smithi153 bash[35780]: cluster 2024-02-10T09:25:43.851781+0000 mgr.y (mgr.14140) 299 : cluster [DBG] pgmap v304: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:43 smithi153 bash[39117]: cluster 2024-02-10T09:25:43.851781+0000 mgr.y (mgr.14140) 299 : cluster [DBG] pgmap v304: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:43 smithi204 bash[35809]: cluster 2024-02-10T09:25:43.851781+0000 mgr.y (mgr.14140) 299 : cluster [DBG] pgmap v304: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:45 smithi153 bash[35780]: cluster 2024-02-10T09:25:45.852289+0000 mgr.y (mgr.14140) 300 : cluster [DBG] pgmap v305: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:45 smithi153 bash[39117]: cluster 2024-02-10T09:25:45.852289+0000 mgr.y (mgr.14140) 300 : cluster [DBG] pgmap v305: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:45 smithi204 bash[35809]: cluster 2024-02-10T09:25:45.852289+0000 mgr.y (mgr.14140) 300 : cluster [DBG] pgmap v305: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:48.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:47 smithi153 bash[35780]: cluster 2024-02-10T09:25:47.852768+0000 mgr.y (mgr.14140) 301 : cluster [DBG] pgmap v306: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:48.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:47 smithi153 bash[39117]: cluster 2024-02-10T09:25:47.852768+0000 mgr.y (mgr.14140) 301 : cluster [DBG] pgmap v306: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:47 smithi204 bash[35809]: cluster 2024-02-10T09:25:47.852768+0000 mgr.y (mgr.14140) 301 : cluster [DBG] pgmap v306: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:49 smithi153 bash[35780]: cluster 2024-02-10T09:25:49.853362+0000 mgr.y (mgr.14140) 302 : cluster [DBG] pgmap v307: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:49 smithi153 bash[39117]: cluster 2024-02-10T09:25:49.853362+0000 mgr.y (mgr.14140) 302 : cluster [DBG] pgmap v307: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:49 smithi204 bash[35809]: cluster 2024-02-10T09:25:49.853362+0000 mgr.y (mgr.14140) 302 : cluster [DBG] pgmap v307: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:52 smithi153 bash[35780]: cluster 2024-02-10T09:25:51.854073+0000 mgr.y (mgr.14140) 303 : cluster [DBG] pgmap v308: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:52 smithi153 bash[39117]: cluster 2024-02-10T09:25:51.854073+0000 mgr.y (mgr.14140) 303 : cluster [DBG] pgmap v308: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:52 smithi204 bash[35809]: cluster 2024-02-10T09:25:51.854073+0000 mgr.y (mgr.14140) 303 : cluster [DBG] pgmap v308: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:53.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T09:25:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:53 smithi153 bash[39117]: cluster 2024-02-10T09:25:53.854831+0000 mgr.y (mgr.14140) 304 : cluster [DBG] pgmap v309: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:53 smithi153 bash[35780]: cluster 2024-02-10T09:25:53.854831+0000 mgr.y (mgr.14140) 304 : cluster [DBG] pgmap v309: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:53 smithi204 bash[35809]: cluster 2024-02-10T09:25:53.854831+0000 mgr.y (mgr.14140) 304 : cluster [DBG] pgmap v309: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:56.162 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:55 smithi153 bash[35780]: cluster 2024-02-10T09:25:55.855279+0000 mgr.y (mgr.14140) 305 : cluster [DBG] pgmap v310: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:56.162 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:55 smithi153 bash[35780]: audit 2024-02-10T09:25:55.864194+0000 mon.a (mon.0) 556 : audit [INF] from='client.? 172.21.15.153:0/3218064128' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim", "value": "false", "force": true}]: dispatch 2024-02-10T09:25:56.163 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:55 smithi153 bash[35780]: audit 2024-02-10T09:25:55.867709+0000 mon.a (mon.0) 557 : audit [INF] from='client.? 172.21.15.153:0/3218064128' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim", "value": "false", "force": true}]': finished 2024-02-10T09:25:56.163 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:55 smithi153 bash[39117]: cluster 2024-02-10T09:25:55.855279+0000 mgr.y (mgr.14140) 305 : cluster [DBG] pgmap v310: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:56.163 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:55 smithi153 bash[39117]: audit 2024-02-10T09:25:55.864194+0000 mon.a (mon.0) 556 : audit [INF] from='client.? 172.21.15.153:0/3218064128' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim", "value": "false", "force": true}]: dispatch 2024-02-10T09:25:56.163 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:55 smithi153 bash[39117]: audit 2024-02-10T09:25:55.867709+0000 mon.a (mon.0) 557 : audit [INF] from='client.? 172.21.15.153:0/3218064128' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim", "value": "false", "force": true}]': finished 2024-02-10T09:25:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:55 smithi204 bash[35809]: cluster 2024-02-10T09:25:55.855279+0000 mgr.y (mgr.14140) 305 : cluster [DBG] pgmap v310: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:55 smithi204 bash[35809]: audit 2024-02-10T09:25:55.864194+0000 mon.a (mon.0) 556 : audit [INF] from='client.? 172.21.15.153:0/3218064128' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim", "value": "false", "force": true}]: dispatch 2024-02-10T09:25:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:55 smithi204 bash[35809]: audit 2024-02-10T09:25:55.867709+0000 mon.a (mon.0) 557 : audit [INF] from='client.? 172.21.15.153:0/3218064128' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim", "value": "false", "force": true}]': finished 2024-02-10T09:25:56.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T09:25:57.967 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:57 smithi153 bash[35780]: cluster 2024-02-10T09:25:57.855742+0000 mgr.y (mgr.14140) 306 : cluster [DBG] pgmap v311: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:57.967 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:57 smithi153 bash[39117]: cluster 2024-02-10T09:25:57.855742+0000 mgr.y (mgr.14140) 306 : cluster [DBG] pgmap v311: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:57 smithi204 bash[35809]: cluster 2024-02-10T09:25:57.855742+0000 mgr.y (mgr.14140) 306 : cluster [DBG] pgmap v311: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:25:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:58 smithi153 bash[35780]: audit 2024-02-10T09:25:58.321547+0000 mon.a (mon.0) 558 : audit [INF] from='client.? 172.21.15.153:0/2206227024' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim_allowed", "value": "false", "force": true}]: dispatch 2024-02-10T09:25:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:58 smithi153 bash[35780]: audit 2024-02-10T09:25:58.326607+0000 mon.a (mon.0) 559 : audit [INF] from='client.? 172.21.15.153:0/2206227024' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim_allowed", "value": "false", "force": true}]': finished 2024-02-10T09:25:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:58 smithi153 bash[39117]: audit 2024-02-10T09:25:58.321547+0000 mon.a (mon.0) 558 : audit [INF] from='client.? 172.21.15.153:0/2206227024' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim_allowed", "value": "false", "force": true}]: dispatch 2024-02-10T09:25:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:58 smithi153 bash[39117]: audit 2024-02-10T09:25:58.326607+0000 mon.a (mon.0) 559 : audit [INF] from='client.? 172.21.15.153:0/2206227024' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim_allowed", "value": "false", "force": true}]': finished 2024-02-10T09:25:59.041 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T09:25:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:58 smithi204 bash[35809]: audit 2024-02-10T09:25:58.321547+0000 mon.a (mon.0) 558 : audit [INF] from='client.? 172.21.15.153:0/2206227024' entity='client.admin' cmd=[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim_allowed", "value": "false", "force": true}]: dispatch 2024-02-10T09:25:59.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:58 smithi204 bash[35809]: audit 2024-02-10T09:25:58.326607+0000 mon.a (mon.0) 559 : audit [INF] from='client.? 172.21.15.153:0/2206227024' entity='client.admin' cmd='[{"prefix": "config set", "who": "mon", "name": "mon_warn_on_insecure_global_id_reclaim_allowed", "value": "false", "force": true}]': finished 2024-02-10T09:26:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:25:59 smithi153 bash[35780]: cluster 2024-02-10T09:25:59.856197+0000 mgr.y (mgr.14140) 307 : cluster [DBG] pgmap v312: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:25:59 smithi153 bash[39117]: cluster 2024-02-10T09:25:59.856197+0000 mgr.y (mgr.14140) 307 : cluster [DBG] pgmap v312: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:25:59 smithi204 bash[35809]: cluster 2024-02-10T09:25:59.856197+0000 mgr.y (mgr.14140) 307 : cluster [DBG] pgmap v312: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:01.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:01 smithi153 bash[39117]: audit 2024-02-10T09:26:01.080172+0000 mon.a (mon.0) 560 : audit [INF] from='client.? 172.21.15.153:0/830090072' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "log_to_journald", "value": "false", "force": true}]: dispatch 2024-02-10T09:26:01.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:01 smithi153 bash[39117]: audit 2024-02-10T09:26:01.083863+0000 mon.a (mon.0) 561 : audit [INF] from='client.? 172.21.15.153:0/830090072' entity='client.admin' cmd='[{"prefix": "config set", "who": "global", "name": "log_to_journald", "value": "false", "force": true}]': finished 2024-02-10T09:26:01.404 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:01 smithi153 bash[35780]: audit 2024-02-10T09:26:01.080172+0000 mon.a (mon.0) 560 : audit [INF] from='client.? 172.21.15.153:0/830090072' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "log_to_journald", "value": "false", "force": true}]: dispatch 2024-02-10T09:26:01.404 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:01 smithi153 bash[35780]: audit 2024-02-10T09:26:01.083863+0000 mon.a (mon.0) 561 : audit [INF] from='client.? 172.21.15.153:0/830090072' entity='client.admin' cmd='[{"prefix": "config set", "who": "global", "name": "log_to_journald", "value": "false", "force": true}]': finished 2024-02-10T09:26:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:01 smithi204 bash[35809]: audit 2024-02-10T09:26:01.080172+0000 mon.a (mon.0) 560 : audit [INF] from='client.? 172.21.15.153:0/830090072' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "log_to_journald", "value": "false", "force": true}]: dispatch 2024-02-10T09:26:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:01 smithi204 bash[35809]: audit 2024-02-10T09:26:01.083863+0000 mon.a (mon.0) 561 : audit [INF] from='client.? 172.21.15.153:0/830090072' entity='client.admin' cmd='[{"prefix": "config set", "who": "global", "name": "log_to_journald", "value": "false", "force": true}]': finished 2024-02-10T09:26:01.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:26:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:02 smithi153 bash[35780]: cluster 2024-02-10T09:26:01.856681+0000 mgr.y (mgr.14140) 308 : cluster [DBG] pgmap v313: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:02 smithi153 bash[39117]: cluster 2024-02-10T09:26:01.856681+0000 mgr.y (mgr.14140) 308 : cluster [DBG] pgmap v313: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:02 smithi204 bash[35809]: cluster 2024-02-10T09:26:01.856681+0000 mgr.y (mgr.14140) 308 : cluster [DBG] pgmap v313: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (7m) 4m ago 7m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 da64912e77c2 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 running (9m) 6m ago 9m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 03548f93e5ac 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (9m) 6m ago 9m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (7m) 4m ago 7m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (8m) 6m ago 8m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (6m) 6m ago 7m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (6m) 6m ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (6m) 6m ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:26:03.546 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (6m) 6m ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:26:03.547 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (5m) 4m ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:26:03.547 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (5m) 4m ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:26:03.547 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (5m) 4m ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:26:03.547 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (4m) 4m ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:26:04.180 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:03 smithi153 bash[35780]: audit 2024-02-10T09:26:03.543322+0000 mgr.y (mgr.14140) 309 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:04.180 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:03 smithi153 bash[35780]: cluster 2024-02-10T09:26:03.857343+0000 mgr.y (mgr.14140) 310 : cluster [DBG] pgmap v314: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:04.181 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:03 smithi153 bash[39117]: audit 2024-02-10T09:26:03.543322+0000 mgr.y (mgr.14140) 309 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:04.181 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:03 smithi153 bash[39117]: cluster 2024-02-10T09:26:03.857343+0000 mgr.y (mgr.14140) 310 : cluster [DBG] pgmap v314: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:04.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:26:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:03 smithi204 bash[35809]: audit 2024-02-10T09:26:03.543322+0000 mgr.y (mgr.14140) 309 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:03 smithi204 bash[35809]: cluster 2024-02-10T09:26:03.857343+0000 mgr.y (mgr.14140) 310 : cluster [DBG] pgmap v314: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:06.141 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:26:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13 2024-02-10T09:26:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:26:06.143 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:26:06.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:05 smithi153 bash[35780]: cluster 2024-02-10T09:26:05.857790+0000 mgr.y (mgr.14140) 311 : cluster [DBG] pgmap v315: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:05 smithi153 bash[39117]: cluster 2024-02-10T09:26:05.857790+0000 mgr.y (mgr.14140) 311 : cluster [DBG] pgmap v315: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:05 smithi204 bash[35809]: cluster 2024-02-10T09:26:05.857790+0000 mgr.y (mgr.14140) 311 : cluster [DBG] pgmap v315: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:06.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph -s' 2024-02-10T09:26:07.098 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:06 smithi153 bash[35780]: audit 2024-02-10T09:26:06.140951+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.153:0/4257644662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:26:07.098 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:06 smithi153 bash[39117]: audit 2024-02-10T09:26:06.140951+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.153:0/4257644662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:26:07.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:06 smithi204 bash[35809]: audit 2024-02-10T09:26:06.140951+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.153:0/4257644662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:26:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:07 smithi153 bash[35780]: cluster 2024-02-10T09:26:07.858253+0000 mgr.y (mgr.14140) 312 : cluster [DBG] pgmap v316: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:07 smithi153 bash[39117]: cluster 2024-02-10T09:26:07.858253+0000 mgr.y (mgr.14140) 312 : cluster [DBG] pgmap v316: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:07 smithi204 bash[35809]: cluster 2024-02-10T09:26:07.858253+0000 mgr.y (mgr.14140) 312 : cluster [DBG] pgmap v316: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: cluster: 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: id: 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: health: HEALTH_OK 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: services: 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: mon: 3 daemons, quorum a,c,b (age 7m) 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: mgr: y(active, since 8m), standbys: x 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: osd: 8 osds: 8 up (since 4m), 8 in (since 4m) 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:26:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: data: 2024-02-10T09:26:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: pools: 5 pools, 129 pgs 2024-02-10T09:26:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: objects: 57 objects, 6.9 KiB 2024-02-10T09:26:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: usage: 8.0 GiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: pgs: 129 active+clean 2024-02-10T09:26:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:26:09.183 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:08 smithi153 bash[39117]: audit 2024-02-10T09:26:08.836481+0000 mon.a (mon.0) 562 : audit [DBG] from='client.? 172.21.15.153:0/3610057393' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:26:09.183 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:08 smithi153 bash[35780]: audit 2024-02-10T09:26:08.836481+0000 mon.a (mon.0) 562 : audit [DBG] from='client.? 172.21.15.153:0/3610057393' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:26:09.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:08 smithi204 bash[35809]: audit 2024-02-10T09:26:08.836481+0000 mon.a (mon.0) 562 : audit [DBG] from='client.? 172.21.15.153:0/3610057393' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:26:09.559 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ls' 2024-02-10T09:26:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:09 smithi153 bash[39117]: cluster 2024-02-10T09:26:09.858723+0000 mgr.y (mgr.14140) 313 : cluster [DBG] pgmap v317: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:09 smithi153 bash[35780]: cluster 2024-02-10T09:26:09.858723+0000 mgr.y (mgr.14140) 313 : cluster [DBG] pgmap v317: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:09 smithi204 bash[35809]: cluster 2024-02-10T09:26:09.858723+0000 mgr.y (mgr.14140) 313 : cluster [DBG] pgmap v317: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:11.493 INFO:teuthology.orchestra.run.smithi153.stdout:NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager 0/1 - - count:1 smithi153=a 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:grafana 0/1 - - count:1 smithi204=a 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:mgr 2/2 6m ago 7m count:2 smithi153=y,smithi204=x docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:mon 3/0 6m ago - docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter 0/2 - - count:2 smithi153=a,smithi204=b 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus 0/1 - - count:1 smithi204=a 2024-02-10T09:26:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z 0/2 - - count:2 2024-02-10T09:26:12.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph config set mgr container_image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T09:26:12.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:12 smithi153 bash[35780]: audit 2024-02-10T09:26:11.491421+0000 mgr.y (mgr.14140) 314 : audit [DBG] from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:12.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:12 smithi153 bash[35780]: cluster 2024-02-10T09:26:11.859219+0000 mgr.y (mgr.14140) 315 : cluster [DBG] pgmap v318: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:12.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:12 smithi153 bash[39117]: audit 2024-02-10T09:26:11.491421+0000 mgr.y (mgr.14140) 314 : audit [DBG] from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:12.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:12 smithi153 bash[39117]: cluster 2024-02-10T09:26:11.859219+0000 mgr.y (mgr.14140) 315 : cluster [DBG] pgmap v318: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:12 smithi204 bash[35809]: audit 2024-02-10T09:26:11.491421+0000 mgr.y (mgr.14140) 314 : audit [DBG] from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:12 smithi204 bash[35809]: cluster 2024-02-10T09:26:11.859219+0000 mgr.y (mgr.14140) 315 : cluster [DBG] pgmap v318: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:13.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[39117]: audit 2024-02-10T09:26:13.634892+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon"}]: dispatch 2024-02-10T09:26:13.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[39117]: audit 2024-02-10T09:26:13.635647+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:13.930 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[35780]: audit 2024-02-10T09:26:13.634892+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon"}]: dispatch 2024-02-10T09:26:13.930 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[35780]: audit 2024-02-10T09:26:13.635647+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:13.930 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: debug 2024-02-10T09:26:13.633+0000 7f90e3a5e700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:13.930 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:26:13.931 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 4247, in 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 968, in _default_image 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 2508, in command_deploy 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 1974, in run 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 696, in call_throws 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: code, '\n'.join(err))) 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:13.932 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 4247, in 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 968, in _default_image 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 2508, in command_deploy 2024-02-10T09:26:13.933 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:13.934 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:13.934 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 1974, in run 2024-02-10T09:26:13.934 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: File "", line 696, in call_throws 2024-02-10T09:26:13.934 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:26:13 smithi153 bash[36014]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:13 smithi204 bash[35809]: audit 2024-02-10T09:26:13.634892+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon"}]: dispatch 2024-02-10T09:26:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:13 smithi204 bash[35809]: audit 2024-02-10T09:26:13.635647+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:14.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:26:14 smithi153 podman[55092]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T09:26:14.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:26:14 smithi153 systemd[1]: Started Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:26:14.669 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[55108]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.669 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[55108]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: cephadm 2024-02-10T09:26:13.634120+0000 mgr.y (mgr.14140) 316 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: cephadm 2024-02-10T09:26:13.634120+0000 mgr.y (mgr.14140) 316 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:26:14.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:26:14.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: cephadm 2024-02-10T09:26:13.634210+0000 mgr.y (mgr.14140) 317 : cephadm [WRN] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi204', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:14.946 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: cephadm 2024-02-10T09:26:13.634210+0000 mgr.y (mgr.14140) 317 : cephadm [WRN] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi204', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.947 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:14.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:14.949 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:26:14.950 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: cephadm 2024-02-10T09:26:13.636368+0000 mgr.y (mgr.14140) 318 : cephadm [INF] Deploying daemon node-exporter.a on smithi153 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: cluster 2024-02-10T09:26:13.859681+0000 mgr.y (mgr.14140) 319 : cluster [DBG] pgmap v319: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: cephadm 2024-02-10T09:26:13.636368+0000 mgr.y (mgr.14140) 318 : cephadm [INF] Deploying daemon node-exporter.a on smithi153 2024-02-10T09:26:14.951 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: cluster 2024-02-10T09:26:13.859681+0000 mgr.y (mgr.14140) 319 : cluster [DBG] pgmap v319: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:14.952 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.190236+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:26:14.952 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.193721+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:26:14.952 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.194327+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon"}]: dispatch 2024-02-10T09:26:14.952 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.194963+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:14.952 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.344851+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.153:0/3807284614' entity='client.admin' cmd=[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]: dispatch 2024-02-10T09:26:14.952 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.190236+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:26:14.953 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.345147+0000 mon.a (mon.0) 569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]: dispatch 2024-02-10T09:26:14.953 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[39117]: audit 2024-02-10T09:26:14.348848+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]': finished 2024-02-10T09:26:14.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.193721+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:26:14.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.194327+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon"}]: dispatch 2024-02-10T09:26:14.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.194963+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:14.953 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.344851+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.153:0/3807284614' entity='client.admin' cmd=[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]: dispatch 2024-02-10T09:26:14.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.345147+0000 mon.a (mon.0) 569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]: dispatch 2024-02-10T09:26:14.954 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:14 smithi153 bash[35780]: audit 2024-02-10T09:26:14.348848+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]': finished 2024-02-10T09:26:14.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)"' 2024-02-10T09:26:15.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:26:14 smithi204 systemd[1]: Started Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:26:15.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[44418]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:15.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[44418]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: cephadm 2024-02-10T09:26:13.634120+0000 mgr.y (mgr.14140) 316 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:15.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:15.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:26:15.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: cephadm 2024-02-10T09:26:13.634210+0000 mgr.y (mgr.14140) 317 : cephadm [WRN] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi204', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:Verifying port 9095 ... 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/prometheus" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:latest... 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:24:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:25:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:latest... 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/prometheus:latest... 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/prometheus:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:latest: reading manifest latest in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/prometheus:latest: reading manifest latest in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/prometheus:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 2448, in extract_uid_gid_monitoring 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:26:15.087 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/prometheus:latest -e NODE_NAME=smithi204 --entrypoint stat prom/prometheus:latest -c %u %g /etc/prometheus 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: cephadm 2024-02-10T09:26:13.636368+0000 mgr.y (mgr.14140) 318 : cephadm [INF] Deploying daemon node-exporter.a on smithi153 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: cluster 2024-02-10T09:26:13.859681+0000 mgr.y (mgr.14140) 319 : cluster [DBG] pgmap v319: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.190236+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.193721+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374279\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374353\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374384\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374411\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374437\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374463\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:19:58.374490\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.194327+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon"}]: dispatch 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.194963+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.344851+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.153:0/3807284614' entity='client.admin' cmd=[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]: dispatch 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.345147+0000 mon.a (mon.0) 569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]: dispatch 2024-02-10T09:26:15.088 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:14 smithi204 bash[35809]: audit 2024-02-10T09:26:14.348848+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config set", "who": "mgr", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1"}]': finished 2024-02-10T09:26:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[35780]: cephadm 2024-02-10T09:26:14.195443+0000 mgr.y (mgr.14140) 320 : cephadm [INF] Deploying daemon node-exporter.b on smithi204 2024-02-10T09:26:15.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[39117]: cephadm 2024-02-10T09:26:14.195443+0000 mgr.y (mgr.14140) 320 : cephadm [INF] Deploying daemon node-exporter.b on smithi204 2024-02-10T09:26:15.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[35780]: audit 2024-02-10T09:26:14.692585+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:26:15.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[35780]: audit 2024-02-10T09:26:14.695203+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:26:15.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[35780]: cephadm 2024-02-10T09:26:14.697283+0000 mgr.y (mgr.14140) 321 : cephadm [INF] Deploying daemon alertmanager.a on smithi153 2024-02-10T09:26:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[39117]: audit 2024-02-10T09:26:14.692585+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:26:15.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[39117]: audit 2024-02-10T09:26:14.695203+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:26:15.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:15 smithi153 bash[39117]: cephadm 2024-02-10T09:26:14.697283+0000 mgr.y (mgr.14140) 321 : cephadm [INF] Deploying daemon alertmanager.a on smithi153 2024-02-10T09:26:16.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:15 smithi204 bash[35809]: cephadm 2024-02-10T09:26:14.195443+0000 mgr.y (mgr.14140) 320 : cephadm [INF] Deploying daemon node-exporter.b on smithi204 2024-02-10T09:26:16.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:15 smithi204 bash[35809]: audit 2024-02-10T09:26:14.692585+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:26:16.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:15 smithi204 bash[35809]: audit 2024-02-10T09:26:14.695203+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"da64912e77c2\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600686\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:18:35.994491\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:34.322208\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:26:16.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:15 smithi204 bash[35809]: cephadm 2024-02-10T09:26:14.697283+0000 mgr.y (mgr.14140) 321 : cephadm [INF] Deploying daemon alertmanager.a on smithi153 2024-02-10T09:26:16.678 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:16 smithi153 bash[35780]: cluster 2024-02-10T09:26:15.860186+0000 mgr.y (mgr.14140) 322 : cluster [DBG] pgmap v320: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:16.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:16 smithi153 bash[39117]: cluster 2024-02-10T09:26:15.860186+0000 mgr.y (mgr.14140) 322 : cluster [DBG] pgmap v320: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:17.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:16 smithi204 bash[35809]: cluster 2024-02-10T09:26:15.860186+0000 mgr.y (mgr.14140) 322 : cluster [DBG] pgmap v320: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:17.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[35780]: audit 2024-02-10T09:26:16.842267+0000 mon.a (mon.0) 573 : audit [DBG] from='client.? 172.21.15.153:0/1121874515' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:26:17.964 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[35780]: audit 2024-02-10T09:26:17.117540+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.x"}]: dispatch 2024-02-10T09:26:17.964 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[35780]: audit 2024-02-10T09:26:17.118074+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:17.964 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[35780]: audit 2024-02-10T09:26:17.118610+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2024-02-10T09:26:17.965 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[39117]: audit 2024-02-10T09:26:16.842267+0000 mon.a (mon.0) 573 : audit [DBG] from='client.? 172.21.15.153:0/1121874515' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:26:17.965 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[39117]: audit 2024-02-10T09:26:17.117540+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.x"}]: dispatch 2024-02-10T09:26:17.965 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[39117]: audit 2024-02-10T09:26:17.118074+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:17.965 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:17 smithi153 bash[39117]: audit 2024-02-10T09:26:17.118610+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2024-02-10T09:26:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:17 smithi204 bash[35809]: audit 2024-02-10T09:26:16.842267+0000 mon.a (mon.0) 573 : audit [DBG] from='client.? 172.21.15.153:0/1121874515' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:26:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:17 smithi204 bash[35809]: audit 2024-02-10T09:26:17.117540+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.x"}]: dispatch 2024-02-10T09:26:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:17 smithi204 bash[35809]: audit 2024-02-10T09:26:17.118074+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:26:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:17 smithi204 bash[35809]: audit 2024-02-10T09:26:17.118610+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[39117]: audit 2024-02-10T09:26:17.116553+0000 mgr.y (mgr.14140) 323 : audit [DBG] from='client.24628 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "redeploy", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[39117]: cephadm 2024-02-10T09:26:17.116922+0000 mgr.y (mgr.14140) 324 : cephadm [INF] Redeploy daemons mgr.x 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[39117]: cephadm 2024-02-10T09:26:17.118433+0000 mgr.y (mgr.14140) 325 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[39117]: cluster 2024-02-10T09:26:17.860566+0000 mgr.y (mgr.14140) 326 : cluster [DBG] pgmap v321: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[35780]: audit 2024-02-10T09:26:17.116553+0000 mgr.y (mgr.14140) 323 : audit [DBG] from='client.24628 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "redeploy", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[35780]: cephadm 2024-02-10T09:26:17.116922+0000 mgr.y (mgr.14140) 324 : cephadm [INF] Redeploy daemons mgr.x 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[35780]: cephadm 2024-02-10T09:26:17.118433+0000 mgr.y (mgr.14140) 325 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:26:18.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:18 smithi153 bash[35780]: cluster 2024-02-10T09:26:17.860566+0000 mgr.y (mgr.14140) 326 : cluster [DBG] pgmap v321: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:18 smithi204 bash[35809]: audit 2024-02-10T09:26:17.116553+0000 mgr.y (mgr.14140) 323 : audit [DBG] from='client.24628 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "redeploy", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:18 smithi204 bash[35809]: cephadm 2024-02-10T09:26:17.116922+0000 mgr.y (mgr.14140) 324 : cephadm [INF] Redeploy daemons mgr.x 2024-02-10T09:26:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:18 smithi204 bash[35809]: cephadm 2024-02-10T09:26:17.118433+0000 mgr.y (mgr.14140) 325 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:26:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:18 smithi204 bash[35809]: cluster 2024-02-10T09:26:17.860566+0000 mgr.y (mgr.14140) 326 : cluster [DBG] pgmap v321: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:20.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:19 smithi153 bash[35780]: cluster 2024-02-10T09:26:19.861109+0000 mgr.y (mgr.14140) 327 : cluster [DBG] pgmap v322: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:19 smithi153 bash[39117]: cluster 2024-02-10T09:26:19.861109+0000 mgr.y (mgr.14140) 327 : cluster [DBG] pgmap v322: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:19 smithi204 bash[35809]: cluster 2024-02-10T09:26:19.861109+0000 mgr.y (mgr.14140) 327 : cluster [DBG] pgmap v322: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:22 smithi153 bash[39117]: cluster 2024-02-10T09:26:21.861785+0000 mgr.y (mgr.14140) 328 : cluster [DBG] pgmap v323: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:22 smithi153 bash[35780]: cluster 2024-02-10T09:26:21.861785+0000 mgr.y (mgr.14140) 328 : cluster [DBG] pgmap v323: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:22 smithi204 bash[35809]: cluster 2024-02-10T09:26:21.861785+0000 mgr.y (mgr.14140) 328 : cluster [DBG] pgmap v323: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:23 smithi153 bash[35780]: cluster 2024-02-10T09:26:23.862327+0000 mgr.y (mgr.14140) 329 : cluster [DBG] pgmap v324: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:23 smithi153 bash[39117]: cluster 2024-02-10T09:26:23.862327+0000 mgr.y (mgr.14140) 329 : cluster [DBG] pgmap v324: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:23 smithi204 bash[35809]: cluster 2024-02-10T09:26:23.862327+0000 mgr.y (mgr.14140) 329 : cluster [DBG] pgmap v324: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:25 smithi153 bash[35780]: cluster 2024-02-10T09:26:25.862811+0000 mgr.y (mgr.14140) 330 : cluster [DBG] pgmap v325: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:25 smithi153 bash[39117]: cluster 2024-02-10T09:26:25.862811+0000 mgr.y (mgr.14140) 330 : cluster [DBG] pgmap v325: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:25 smithi204 bash[35809]: cluster 2024-02-10T09:26:25.862811+0000 mgr.y (mgr.14140) 330 : cluster [DBG] pgmap v325: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:27 smithi153 bash[35780]: cluster 2024-02-10T09:26:27.863312+0000 mgr.y (mgr.14140) 331 : cluster [DBG] pgmap v326: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:27 smithi153 bash[39117]: cluster 2024-02-10T09:26:27.863312+0000 mgr.y (mgr.14140) 331 : cluster [DBG] pgmap v326: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:27 smithi204 bash[35809]: cluster 2024-02-10T09:26:27.863312+0000 mgr.y (mgr.14140) 331 : cluster [DBG] pgmap v326: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:29 smithi153 bash[35780]: cluster 2024-02-10T09:26:29.864019+0000 mgr.y (mgr.14140) 332 : cluster [DBG] pgmap v327: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:29 smithi153 bash[39117]: cluster 2024-02-10T09:26:29.864019+0000 mgr.y (mgr.14140) 332 : cluster [DBG] pgmap v327: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:29 smithi204 bash[35809]: cluster 2024-02-10T09:26:29.864019+0000 mgr.y (mgr.14140) 332 : cluster [DBG] pgmap v327: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:32 smithi153 bash[39117]: cluster 2024-02-10T09:26:31.864568+0000 mgr.y (mgr.14140) 333 : cluster [DBG] pgmap v328: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:32 smithi153 bash[35780]: cluster 2024-02-10T09:26:31.864568+0000 mgr.y (mgr.14140) 333 : cluster [DBG] pgmap v328: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:32 smithi204 bash[35809]: cluster 2024-02-10T09:26:31.864568+0000 mgr.y (mgr.14140) 333 : cluster [DBG] pgmap v328: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:33 smithi153 bash[35780]: cluster 2024-02-10T09:26:33.865084+0000 mgr.y (mgr.14140) 334 : cluster [DBG] pgmap v329: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:34.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:33 smithi153 bash[39117]: cluster 2024-02-10T09:26:33.865084+0000 mgr.y (mgr.14140) 334 : cluster [DBG] pgmap v329: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:33 smithi204 bash[35809]: cluster 2024-02-10T09:26:33.865084+0000 mgr.y (mgr.14140) 334 : cluster [DBG] pgmap v329: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:35 smithi153 bash[35780]: cluster 2024-02-10T09:26:35.865522+0000 mgr.y (mgr.14140) 335 : cluster [DBG] pgmap v330: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:35 smithi153 bash[39117]: cluster 2024-02-10T09:26:35.865522+0000 mgr.y (mgr.14140) 335 : cluster [DBG] pgmap v330: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:35 smithi204 bash[35809]: cluster 2024-02-10T09:26:35.865522+0000 mgr.y (mgr.14140) 335 : cluster [DBG] pgmap v330: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:37 smithi153 bash[35780]: cluster 2024-02-10T09:26:37.865945+0000 mgr.y (mgr.14140) 336 : cluster [DBG] pgmap v331: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:37 smithi153 bash[39117]: cluster 2024-02-10T09:26:37.865945+0000 mgr.y (mgr.14140) 336 : cluster [DBG] pgmap v331: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:37 smithi204 bash[35809]: cluster 2024-02-10T09:26:37.865945+0000 mgr.y (mgr.14140) 336 : cluster [DBG] pgmap v331: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:39 smithi153 bash[35780]: cluster 2024-02-10T09:26:39.866483+0000 mgr.y (mgr.14140) 337 : cluster [DBG] pgmap v332: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:39 smithi153 bash[39117]: cluster 2024-02-10T09:26:39.866483+0000 mgr.y (mgr.14140) 337 : cluster [DBG] pgmap v332: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:39 smithi204 bash[35809]: cluster 2024-02-10T09:26:39.866483+0000 mgr.y (mgr.14140) 337 : cluster [DBG] pgmap v332: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:42.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:42 smithi153 bash[35780]: cluster 2024-02-10T09:26:41.867125+0000 mgr.y (mgr.14140) 338 : cluster [DBG] pgmap v333: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:42.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:42 smithi153 bash[39117]: cluster 2024-02-10T09:26:41.867125+0000 mgr.y (mgr.14140) 338 : cluster [DBG] pgmap v333: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:42 smithi204 bash[35809]: cluster 2024-02-10T09:26:41.867125+0000 mgr.y (mgr.14140) 338 : cluster [DBG] pgmap v333: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:43 smithi153 bash[35780]: cluster 2024-02-10T09:26:43.867840+0000 mgr.y (mgr.14140) 339 : cluster [DBG] pgmap v334: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:43 smithi153 bash[39117]: cluster 2024-02-10T09:26:43.867840+0000 mgr.y (mgr.14140) 339 : cluster [DBG] pgmap v334: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:44.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:43 smithi204 bash[35809]: cluster 2024-02-10T09:26:43.867840+0000 mgr.y (mgr.14140) 339 : cluster [DBG] pgmap v334: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:45.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:45 smithi204 systemd[1]: Stopping Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:26:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:45 smithi153 bash[35780]: cluster 2024-02-10T09:26:45.868460+0000 mgr.y (mgr.14140) 340 : cluster [DBG] pgmap v335: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:45 smithi153 bash[39117]: cluster 2024-02-10T09:26:45.868460+0000 mgr.y (mgr.14140) 340 : cluster [DBG] pgmap v335: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:45 smithi204 bash[35809]: cluster 2024-02-10T09:26:45.868460+0000 mgr.y (mgr.14140) 340 : cluster [DBG] pgmap v335: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:47 smithi153 bash[35780]: cluster 2024-02-10T09:26:47.868941+0000 mgr.y (mgr.14140) 341 : cluster [DBG] pgmap v336: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:47 smithi153 bash[39117]: cluster 2024-02-10T09:26:47.868941+0000 mgr.y (mgr.14140) 341 : cluster [DBG] pgmap v336: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:47 smithi204 bash[35809]: cluster 2024-02-10T09:26:47.868941+0000 mgr.y (mgr.14140) 341 : cluster [DBG] pgmap v336: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:49 smithi153 bash[35780]: cluster 2024-02-10T09:26:49.869593+0000 mgr.y (mgr.14140) 342 : cluster [DBG] pgmap v337: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:49 smithi153 bash[39117]: cluster 2024-02-10T09:26:49.869593+0000 mgr.y (mgr.14140) 342 : cluster [DBG] pgmap v337: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:49 smithi204 bash[35809]: cluster 2024-02-10T09:26:49.869593+0000 mgr.y (mgr.14140) 342 : cluster [DBG] pgmap v337: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:51.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:26:50 smithi153 bash[55108]: time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:51.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:26:51 smithi204 bash[44418]: time="2024-02-10T09:26:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:26:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:52 smithi153 bash[35780]: cluster 2024-02-10T09:26:51.870111+0000 mgr.y (mgr.14140) 343 : cluster [DBG] pgmap v338: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:52 smithi153 bash[39117]: cluster 2024-02-10T09:26:51.870111+0000 mgr.y (mgr.14140) 343 : cluster [DBG] pgmap v338: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:52 smithi204 bash[35809]: cluster 2024-02-10T09:26:51.870111+0000 mgr.y (mgr.14140) 343 : cluster [DBG] pgmap v338: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:53 smithi153 bash[35780]: cluster 2024-02-10T09:26:53.870679+0000 mgr.y (mgr.14140) 344 : cluster [DBG] pgmap v339: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:53 smithi153 bash[39117]: cluster 2024-02-10T09:26:53.870679+0000 mgr.y (mgr.14140) 344 : cluster [DBG] pgmap v339: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:53 smithi204 bash[35809]: cluster 2024-02-10T09:26:53.870679+0000 mgr.y (mgr.14140) 344 : cluster [DBG] pgmap v339: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:55.646 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:55 smithi204 podman[44550]: time="2024-02-10T09:26:55Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.x in 10 seconds, resorting to SIGKILL" 2024-02-10T09:26:55.931 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:55 smithi204 podman[44550]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.x 2024-02-10T09:26:56.177 INFO:teuthology.orchestra.run.smithi153.stdout:Deployed mgr.x on host 'smithi204' 2024-02-10T09:26:56.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:55 smithi153 bash[39117]: cluster 2024-02-10T09:26:55.871189+0000 mgr.y (mgr.14140) 345 : cluster [DBG] pgmap v340: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:55 smithi153 bash[35780]: cluster 2024-02-10T09:26:55.871189+0000 mgr.y (mgr.14140) 345 : cluster [DBG] pgmap v340: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:56.308 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:55 smithi204 bash[35809]: cluster 2024-02-10T09:26:55.871189+0000 mgr.y (mgr.14140) 345 : cluster [DBG] pgmap v340: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:56.308 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=137/n/a 2024-02-10T09:26:56.308 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T09:26:56.309 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:55 smithi204 systemd[1]: Stopped Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:26:56.309 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:56 smithi204 systemd[1]: Starting Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:26:56.309 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:56 smithi204 podman[44629]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T09:26:56.309 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:56 smithi204 systemd[1]: Started Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:26:56.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps --refresh' 2024-02-10T09:26:56.935 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:56 smithi204 bash[44646]: debug 2024-02-10T09:26:56.499+0000 7f68a8555700 1 -- 172.21.15.204:0/3381472988 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5576d48f64e0 con 0x5576d48cb000 2024-02-10T09:26:56.935 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:56 smithi204 bash[44646]: debug 2024-02-10T09:26:56.620+0000 7f68b8b51680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:26:56.935 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:56 smithi204 bash[44646]: debug 2024-02-10T09:26:56.700+0000 7f68b8b51680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:26:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:56 smithi153 bash[35780]: audit 2024-02-10T09:26:56.125790+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:26:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:56 smithi153 bash[35780]: audit 2024-02-10T09:26:56.131025+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:26:57.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:56 smithi153 bash[39117]: audit 2024-02-10T09:26:56.125790+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:26:57.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:56 smithi153 bash[39117]: audit 2024-02-10T09:26:56.131025+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:26:57.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:57 smithi204 bash[44646]: debug 2024-02-10T09:26:57.064+0000 7f68b8b51680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:26:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:56 smithi204 bash[35809]: audit 2024-02-10T09:26:56.125790+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:26:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:56 smithi204 bash[35809]: audit 2024-02-10T09:26:56.131025+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600612\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600717\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600755\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600783\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:21:08.600810\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:26:58.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:57 smithi204 bash[35809]: cluster 2024-02-10T09:26:57.871815+0000 mgr.y (mgr.14140) 346 : cluster [DBG] pgmap v341: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:58.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:57 smithi204 bash[44646]: debug 2024-02-10T09:26:57.807+0000 7f68b8b51680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:26:58.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:57 smithi204 bash[44646]: debug 2024-02-10T09:26:57.893+0000 7f68b8b51680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:26:58.132 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:57 smithi153 bash[35780]: cluster 2024-02-10T09:26:57.871815+0000 mgr.y (mgr.14140) 346 : cluster [DBG] pgmap v341: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:58.132 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:57 smithi153 bash[39117]: cluster 2024-02-10T09:26:57.871815+0000 mgr.y (mgr.14140) 346 : cluster [DBG] pgmap v341: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:26:58.583 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:58 smithi204 bash[44646]: debug 2024-02-10T09:26:58.106+0000 7f68b8b51680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:26:59.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:58 smithi204 bash[35809]: audit 2024-02-10T09:26:58.735414+0000 mgr.y (mgr.14140) 347 : audit [DBG] from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:59.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:58 smithi204 bash[35809]: audit 2024-02-10T09:26:58.735965+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:26:59.270 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.061+0000 7f68b8b51680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:26:59.270 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.190+0000 7f68b8b51680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:26:59.438 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:58 smithi153 bash[35780]: audit 2024-02-10T09:26:58.735414+0000 mgr.y (mgr.14140) 347 : audit [DBG] from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:59.438 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:58 smithi153 bash[35780]: audit 2024-02-10T09:26:58.735965+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:26:59.438 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:58 smithi153 bash[39117]: audit 2024-02-10T09:26:58.735414+0000 mgr.y (mgr.14140) 347 : audit [DBG] from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:26:59.438 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:58 smithi153 bash[39117]: audit 2024-02-10T09:26:58.735965+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:26:59.554 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.268+0000 7f68b8b51680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:26:59.554 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.357+0000 7f68b8b51680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:26:59.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.553+0000 7f68b8b51680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:27:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[35809]: cluster 2024-02-10T09:26:59.872260+0000 mgr.y (mgr.14140) 348 : cluster [DBG] pgmap v342: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:00.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.831+0000 7f68b8b51680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:27:00.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:26:59 smithi204 bash[44646]: debug 2024-02-10T09:26:59.929+0000 7f68b8b51680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:27:00.438 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:26:59 smithi153 bash[35780]: cluster 2024-02-10T09:26:59.872260+0000 mgr.y (mgr.14140) 348 : cluster [DBG] pgmap v342: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:00.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:26:59 smithi153 bash[39117]: cluster 2024-02-10T09:26:59.872260+0000 mgr.y (mgr.14140) 348 : cluster [DBG] pgmap v342: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:00.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:00 smithi204 bash[44646]: debug 2024-02-10T09:27:00.688+0000 7f68b8b51680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:27:00.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:00 smithi204 bash[44646]: debug 2024-02-10T09:27:00.773+0000 7f68b8b51680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:27:01.312 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:00 smithi204 bash[44646]: debug 2024-02-10T09:27:00.867+0000 7f68b8b51680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:27:01.312 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:01 smithi204 bash[44646]: debug 2024-02-10T09:27:01.042+0000 7f68b8b51680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:27:01.313 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:00 smithi204 bash[35809]: audit 2024-02-10T09:27:00.097981+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_daemon_update\": \"2024-02-10T09:27:00.097271\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:27:01.313 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:00 smithi204 bash[35809]: audit 2024-02-10T09:27:00.103099+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_daemon_update\": \"2024-02-10T09:27:00.097271\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:27:01.313 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:00 smithi204 bash[35809]: audit 2024-02-10T09:27:00.103535+0000 mon.a (mon.0) 582 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:27:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:00 smithi153 bash[35780]: audit 2024-02-10T09:27:00.097981+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_daemon_update\": \"2024-02-10T09:27:00.097271\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:27:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:00 smithi153 bash[39117]: audit 2024-02-10T09:27:00.097981+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_daemon_update\": \"2024-02-10T09:27:00.097271\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:27:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:00 smithi153 bash[39117]: audit 2024-02-10T09:27:00.103099+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_daemon_update\": \"2024-02-10T09:27:00.097271\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:27:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:00 smithi153 bash[39117]: audit 2024-02-10T09:27:00.103535+0000 mon.a (mon.0) 582 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:27:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:00 smithi153 bash[35780]: audit 2024-02-10T09:27:00.103099+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}}, \"last_daemon_update\": \"2024-02-10T09:27:00.097271\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:27:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:00 smithi153 bash[35780]: audit 2024-02-10T09:27:00.103535+0000 mon.a (mon.0) 582 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:27:01.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:01 smithi204 bash[44646]: debug 2024-02-10T09:27:01.310+0000 7f68b8b51680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:27:01.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:01 smithi204 bash[44646]: debug 2024-02-10T09:27:01.449+0000 7f68b8b51680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:27:01.856 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:01 smithi204 bash[44646]: debug 2024-02-10T09:27:01.587+0000 7f68b8b51680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:27:01.857 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:01 smithi204 bash[44646]: debug 2024-02-10T09:27:01.762+0000 7f68b8b51680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:27:02.185 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID 2024-02-10T09:27:02.185 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (6s) 0s ago 8m 16.2.14-521-g0e714d9a quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 1fbc92c859dc 46c6073a79be 2024-02-10T09:27:02.185 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (8m) 0s ago 8m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:27:02.185 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running 0s ago 47s prom/node-exporter 2024-02-10T09:27:02.186 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (6m) 0s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:27:02.186 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (6m) 0s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:27:02.186 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (6m) 0s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:27:02.186 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (5m) 0s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:27:02.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:27:01 smithi204 bash[44646]: debug 2024-02-10T09:27:01.855+0000 7f68b8b51680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:27:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: cluster 2024-02-10T09:27:01.869983+0000 mon.a (mon.0) 583 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:27:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: cluster 2024-02-10T09:27:01.870094+0000 mon.a (mon.0) 584 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:27:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: audit 2024-02-10T09:27:01.871195+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:27:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: audit 2024-02-10T09:27:01.871529+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: audit 2024-02-10T09:27:01.872689+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: cluster 2024-02-10T09:27:01.872722+0000 mgr.y (mgr.14140) 349 : cluster [DBG] pgmap v343: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: audit 2024-02-10T09:27:01.872957+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: cluster 2024-02-10T09:27:01.869983+0000 mon.a (mon.0) 583 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: cluster 2024-02-10T09:27:01.870094+0000 mon.a (mon.0) 584 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: audit 2024-02-10T09:27:01.871195+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: audit 2024-02-10T09:27:01.871529+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:27:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: audit 2024-02-10T09:27:01.872689+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:27:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: cluster 2024-02-10T09:27:01.872722+0000 mgr.y (mgr.14140) 349 : cluster [DBG] pgmap v343: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: audit 2024-02-10T09:27:01.872957+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:27:02.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: audit 2024-02-10T09:27:02.179151+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_daemon_update\": \"2024-02-10T09:27:02.178402\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:27:02.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[35780]: audit 2024-02-10T09:27:02.182358+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_daemon_update\": \"2024-02-10T09:27:02.178402\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:27:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: audit 2024-02-10T09:27:02.179151+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_daemon_update\": \"2024-02-10T09:27:02.178402\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:27:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:02 smithi153 bash[39117]: audit 2024-02-10T09:27:02.182358+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_daemon_update\": \"2024-02-10T09:27:02.178402\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:27:02.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 180' 2024-02-10T09:27:03.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: cluster 2024-02-10T09:27:01.869983+0000 mon.a (mon.0) 583 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: cluster 2024-02-10T09:27:01.870094+0000 mon.a (mon.0) 584 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: audit 2024-02-10T09:27:01.871195+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: audit 2024-02-10T09:27:01.871529+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: audit 2024-02-10T09:27:01.872689+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: cluster 2024-02-10T09:27:01.872722+0000 mgr.y (mgr.14140) 349 : cluster [DBG] pgmap v343: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: audit 2024-02-10T09:27:01.872957+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: audit 2024-02-10T09:27:02.179151+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_daemon_update\": \"2024-02-10T09:27:02.178402\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:27:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:02 smithi204 bash[35809]: audit 2024-02-10T09:27:02.182358+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}}, \"last_daemon_update\": \"2024-02-10T09:27:02.178402\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:27:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:03 smithi153 bash[35780]: cluster 2024-02-10T09:27:02.674334+0000 mon.a (mon.0) 587 : cluster [DBG] mgrmap e15: y(active, since 9m), standbys: x 2024-02-10T09:27:03.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:03 smithi153 bash[39117]: cluster 2024-02-10T09:27:02.674334+0000 mon.a (mon.0) 587 : cluster [DBG] mgrmap e15: y(active, since 9m), standbys: x 2024-02-10T09:27:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:03 smithi204 bash[35809]: cluster 2024-02-10T09:27:02.674334+0000 mon.a (mon.0) 587 : cluster [DBG] mgrmap e15: y(active, since 9m), standbys: x 2024-02-10T09:27:04.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:04 smithi153 bash[35780]: cluster 2024-02-10T09:27:03.873202+0000 mgr.y (mgr.14140) 350 : cluster [DBG] pgmap v344: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:04.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:04 smithi153 bash[39117]: cluster 2024-02-10T09:27:03.873202+0000 mgr.y (mgr.14140) 350 : cluster [DBG] pgmap v344: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:05.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:04 smithi204 bash[35809]: cluster 2024-02-10T09:27:03.873202+0000 mgr.y (mgr.14140) 350 : cluster [DBG] pgmap v344: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:05 smithi153 bash[35780]: cluster 2024-02-10T09:27:05.873695+0000 mgr.y (mgr.14140) 351 : cluster [DBG] pgmap v345: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:05 smithi153 bash[39117]: cluster 2024-02-10T09:27:05.873695+0000 mgr.y (mgr.14140) 351 : cluster [DBG] pgmap v345: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:05 smithi204 bash[35809]: cluster 2024-02-10T09:27:05.873695+0000 mgr.y (mgr.14140) 351 : cluster [DBG] pgmap v345: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:07 smithi153 bash[35780]: cluster 2024-02-10T09:27:07.874273+0000 mgr.y (mgr.14140) 352 : cluster [DBG] pgmap v346: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:07 smithi153 bash[39117]: cluster 2024-02-10T09:27:07.874273+0000 mgr.y (mgr.14140) 352 : cluster [DBG] pgmap v346: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:07 smithi204 bash[35809]: cluster 2024-02-10T09:27:07.874273+0000 mgr.y (mgr.14140) 352 : cluster [DBG] pgmap v346: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:09 smithi153 bash[35780]: cluster 2024-02-10T09:27:09.874789+0000 mgr.y (mgr.14140) 353 : cluster [DBG] pgmap v347: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:09 smithi153 bash[39117]: cluster 2024-02-10T09:27:09.874789+0000 mgr.y (mgr.14140) 353 : cluster [DBG] pgmap v347: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:09 smithi204 bash[35809]: cluster 2024-02-10T09:27:09.874789+0000 mgr.y (mgr.14140) 353 : cluster [DBG] pgmap v347: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:12.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:12 smithi153 bash[35780]: cluster 2024-02-10T09:27:11.875341+0000 mgr.y (mgr.14140) 354 : cluster [DBG] pgmap v348: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:12.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:12 smithi153 bash[39117]: cluster 2024-02-10T09:27:11.875341+0000 mgr.y (mgr.14140) 354 : cluster [DBG] pgmap v348: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:12 smithi204 bash[35809]: cluster 2024-02-10T09:27:11.875341+0000 mgr.y (mgr.14140) 354 : cluster [DBG] pgmap v348: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:13 smithi153 bash[35780]: cluster 2024-02-10T09:27:13.876017+0000 mgr.y (mgr.14140) 355 : cluster [DBG] pgmap v349: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:13 smithi153 bash[39117]: cluster 2024-02-10T09:27:13.876017+0000 mgr.y (mgr.14140) 355 : cluster [DBG] pgmap v349: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:13 smithi204 bash[35809]: cluster 2024-02-10T09:27:13.876017+0000 mgr.y (mgr.14140) 355 : cluster [DBG] pgmap v349: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:15 smithi153 bash[35780]: cluster 2024-02-10T09:27:15.876499+0000 mgr.y (mgr.14140) 356 : cluster [DBG] pgmap v350: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:15 smithi153 bash[39117]: cluster 2024-02-10T09:27:15.876499+0000 mgr.y (mgr.14140) 356 : cluster [DBG] pgmap v350: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:15 smithi204 bash[35809]: cluster 2024-02-10T09:27:15.876499+0000 mgr.y (mgr.14140) 356 : cluster [DBG] pgmap v350: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:17 smithi153 bash[35780]: cluster 2024-02-10T09:27:17.877078+0000 mgr.y (mgr.14140) 357 : cluster [DBG] pgmap v351: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:17 smithi153 bash[39117]: cluster 2024-02-10T09:27:17.877078+0000 mgr.y (mgr.14140) 357 : cluster [DBG] pgmap v351: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:17 smithi204 bash[35809]: cluster 2024-02-10T09:27:17.877078+0000 mgr.y (mgr.14140) 357 : cluster [DBG] pgmap v351: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:19 smithi153 bash[35780]: cluster 2024-02-10T09:27:19.877640+0000 mgr.y (mgr.14140) 358 : cluster [DBG] pgmap v352: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:19 smithi153 bash[39117]: cluster 2024-02-10T09:27:19.877640+0000 mgr.y (mgr.14140) 358 : cluster [DBG] pgmap v352: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:19 smithi204 bash[35809]: cluster 2024-02-10T09:27:19.877640+0000 mgr.y (mgr.14140) 358 : cluster [DBG] pgmap v352: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:22 smithi153 bash[35780]: cluster 2024-02-10T09:27:21.878203+0000 mgr.y (mgr.14140) 359 : cluster [DBG] pgmap v353: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:22 smithi153 bash[39117]: cluster 2024-02-10T09:27:21.878203+0000 mgr.y (mgr.14140) 359 : cluster [DBG] pgmap v353: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:22 smithi204 bash[35809]: cluster 2024-02-10T09:27:21.878203+0000 mgr.y (mgr.14140) 359 : cluster [DBG] pgmap v353: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:23 smithi153 bash[35780]: cluster 2024-02-10T09:27:23.878879+0000 mgr.y (mgr.14140) 360 : cluster [DBG] pgmap v354: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:23 smithi153 bash[39117]: cluster 2024-02-10T09:27:23.878879+0000 mgr.y (mgr.14140) 360 : cluster [DBG] pgmap v354: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:23 smithi204 bash[35809]: cluster 2024-02-10T09:27:23.878879+0000 mgr.y (mgr.14140) 360 : cluster [DBG] pgmap v354: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:25 smithi153 bash[35780]: cluster 2024-02-10T09:27:25.879487+0000 mgr.y (mgr.14140) 361 : cluster [DBG] pgmap v355: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:25 smithi153 bash[39117]: cluster 2024-02-10T09:27:25.879487+0000 mgr.y (mgr.14140) 361 : cluster [DBG] pgmap v355: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:25 smithi204 bash[35809]: cluster 2024-02-10T09:27:25.879487+0000 mgr.y (mgr.14140) 361 : cluster [DBG] pgmap v355: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:27 smithi204 bash[35809]: cluster 2024-02-10T09:27:27.880146+0000 mgr.y (mgr.14140) 362 : cluster [DBG] pgmap v356: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:28.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:27 smithi153 bash[35780]: cluster 2024-02-10T09:27:27.880146+0000 mgr.y (mgr.14140) 362 : cluster [DBG] pgmap v356: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:28.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:27 smithi153 bash[39117]: cluster 2024-02-10T09:27:27.880146+0000 mgr.y (mgr.14140) 362 : cluster [DBG] pgmap v356: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:28.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:27:28 smithi153 bash[55108]: time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:29.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:27:28 smithi204 bash[44418]: time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:29 smithi204 bash[35809]: cluster 2024-02-10T09:27:29.880774+0000 mgr.y (mgr.14140) 363 : cluster [DBG] pgmap v357: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:30.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:29 smithi153 bash[35780]: cluster 2024-02-10T09:27:29.880774+0000 mgr.y (mgr.14140) 363 : cluster [DBG] pgmap v357: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:30.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:29 smithi153 bash[39117]: cluster 2024-02-10T09:27:29.880774+0000 mgr.y (mgr.14140) 363 : cluster [DBG] pgmap v357: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:32 smithi153 bash[35780]: cluster 2024-02-10T09:27:31.881272+0000 mgr.y (mgr.14140) 364 : cluster [DBG] pgmap v358: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:32 smithi153 bash[39117]: cluster 2024-02-10T09:27:31.881272+0000 mgr.y (mgr.14140) 364 : cluster [DBG] pgmap v358: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:32 smithi204 bash[35809]: cluster 2024-02-10T09:27:31.881272+0000 mgr.y (mgr.14140) 364 : cluster [DBG] pgmap v358: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:33 smithi153 bash[35780]: cluster 2024-02-10T09:27:33.881829+0000 mgr.y (mgr.14140) 365 : cluster [DBG] pgmap v359: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:33 smithi153 bash[39117]: cluster 2024-02-10T09:27:33.881829+0000 mgr.y (mgr.14140) 365 : cluster [DBG] pgmap v359: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:33 smithi204 bash[35809]: cluster 2024-02-10T09:27:33.881829+0000 mgr.y (mgr.14140) 365 : cluster [DBG] pgmap v359: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:37.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:37 smithi153 bash[35780]: cluster 2024-02-10T09:27:35.882408+0000 mgr.y (mgr.14140) 366 : cluster [DBG] pgmap v360: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:37.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:37 smithi153 bash[39117]: cluster 2024-02-10T09:27:35.882408+0000 mgr.y (mgr.14140) 366 : cluster [DBG] pgmap v360: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:37 smithi204 bash[35809]: cluster 2024-02-10T09:27:35.882408+0000 mgr.y (mgr.14140) 366 : cluster [DBG] pgmap v360: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:39 smithi153 bash[35780]: cluster 2024-02-10T09:27:37.882867+0000 mgr.y (mgr.14140) 367 : cluster [DBG] pgmap v361: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:39.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:39 smithi153 bash[39117]: cluster 2024-02-10T09:27:37.882867+0000 mgr.y (mgr.14140) 367 : cluster [DBG] pgmap v361: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:39 smithi204 bash[35809]: cluster 2024-02-10T09:27:37.882867+0000 mgr.y (mgr.14140) 367 : cluster [DBG] pgmap v361: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:41.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:41 smithi153 bash[35780]: cluster 2024-02-10T09:27:39.883427+0000 mgr.y (mgr.14140) 368 : cluster [DBG] pgmap v362: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:41.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:41 smithi153 bash[39117]: cluster 2024-02-10T09:27:39.883427+0000 mgr.y (mgr.14140) 368 : cluster [DBG] pgmap v362: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:41 smithi204 bash[35809]: cluster 2024-02-10T09:27:39.883427+0000 mgr.y (mgr.14140) 368 : cluster [DBG] pgmap v362: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:43.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:43 smithi153 bash[35780]: cluster 2024-02-10T09:27:41.884030+0000 mgr.y (mgr.14140) 369 : cluster [DBG] pgmap v363: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:43.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:43 smithi153 bash[39117]: cluster 2024-02-10T09:27:41.884030+0000 mgr.y (mgr.14140) 369 : cluster [DBG] pgmap v363: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:43 smithi204 bash[35809]: cluster 2024-02-10T09:27:41.884030+0000 mgr.y (mgr.14140) 369 : cluster [DBG] pgmap v363: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:45 smithi204 bash[35809]: cluster 2024-02-10T09:27:43.884618+0000 mgr.y (mgr.14140) 370 : cluster [DBG] pgmap v364: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:45 smithi153 bash[35780]: cluster 2024-02-10T09:27:43.884618+0000 mgr.y (mgr.14140) 370 : cluster [DBG] pgmap v364: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:45 smithi153 bash[39117]: cluster 2024-02-10T09:27:43.884618+0000 mgr.y (mgr.14140) 370 : cluster [DBG] pgmap v364: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:48.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:47 smithi204 bash[35809]: cluster 2024-02-10T09:27:45.885238+0000 mgr.y (mgr.14140) 371 : cluster [DBG] pgmap v365: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:47 smithi153 bash[35780]: cluster 2024-02-10T09:27:45.885238+0000 mgr.y (mgr.14140) 371 : cluster [DBG] pgmap v365: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:47 smithi153 bash[39117]: cluster 2024-02-10T09:27:45.885238+0000 mgr.y (mgr.14140) 371 : cluster [DBG] pgmap v365: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:49 smithi204 bash[35809]: cluster 2024-02-10T09:27:47.885871+0000 mgr.y (mgr.14140) 372 : cluster [DBG] pgmap v366: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:49 smithi153 bash[35780]: cluster 2024-02-10T09:27:47.885871+0000 mgr.y (mgr.14140) 372 : cluster [DBG] pgmap v366: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:49 smithi153 bash[39117]: cluster 2024-02-10T09:27:47.885871+0000 mgr.y (mgr.14140) 372 : cluster [DBG] pgmap v366: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:51 smithi204 bash[35809]: cluster 2024-02-10T09:27:49.886375+0000 mgr.y (mgr.14140) 373 : cluster [DBG] pgmap v367: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:52.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:51 smithi153 bash[35780]: cluster 2024-02-10T09:27:49.886375+0000 mgr.y (mgr.14140) 373 : cluster [DBG] pgmap v367: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:52.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:51 smithi153 bash[39117]: cluster 2024-02-10T09:27:49.886375+0000 mgr.y (mgr.14140) 373 : cluster [DBG] pgmap v367: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:53 smithi204 bash[35809]: cluster 2024-02-10T09:27:51.887094+0000 mgr.y (mgr.14140) 374 : cluster [DBG] pgmap v368: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:53 smithi153 bash[35780]: cluster 2024-02-10T09:27:51.887094+0000 mgr.y (mgr.14140) 374 : cluster [DBG] pgmap v368: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:53 smithi153 bash[39117]: cluster 2024-02-10T09:27:51.887094+0000 mgr.y (mgr.14140) 374 : cluster [DBG] pgmap v368: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:56.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:55 smithi204 bash[35809]: cluster 2024-02-10T09:27:53.887800+0000 mgr.y (mgr.14140) 375 : cluster [DBG] pgmap v369: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:55 smithi153 bash[35780]: cluster 2024-02-10T09:27:53.887800+0000 mgr.y (mgr.14140) 375 : cluster [DBG] pgmap v369: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:55 smithi153 bash[39117]: cluster 2024-02-10T09:27:53.887800+0000 mgr.y (mgr.14140) 375 : cluster [DBG] pgmap v369: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:58.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:57 smithi204 bash[35809]: cluster 2024-02-10T09:27:55.888334+0000 mgr.y (mgr.14140) 376 : cluster [DBG] pgmap v370: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:58.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:57 smithi153 bash[35780]: cluster 2024-02-10T09:27:55.888334+0000 mgr.y (mgr.14140) 376 : cluster [DBG] pgmap v370: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:27:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:57 smithi153 bash[39117]: cluster 2024-02-10T09:27:55.888334+0000 mgr.y (mgr.14140) 376 : cluster [DBG] pgmap v370: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:27:59 smithi204 bash[35809]: cluster 2024-02-10T09:27:57.888942+0000 mgr.y (mgr.14140) 377 : cluster [DBG] pgmap v371: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:27:59 smithi153 bash[35780]: cluster 2024-02-10T09:27:57.888942+0000 mgr.y (mgr.14140) 377 : cluster [DBG] pgmap v371: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:27:59 smithi153 bash[39117]: cluster 2024-02-10T09:27:57.888942+0000 mgr.y (mgr.14140) 377 : cluster [DBG] pgmap v371: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:01 smithi204 bash[35809]: cluster 2024-02-10T09:27:59.889435+0000 mgr.y (mgr.14140) 378 : cluster [DBG] pgmap v372: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:01 smithi153 bash[35780]: cluster 2024-02-10T09:27:59.889435+0000 mgr.y (mgr.14140) 378 : cluster [DBG] pgmap v372: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:01 smithi153 bash[39117]: cluster 2024-02-10T09:27:59.889435+0000 mgr.y (mgr.14140) 378 : cluster [DBG] pgmap v372: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:03 smithi204 bash[35809]: cluster 2024-02-10T09:28:01.890012+0000 mgr.y (mgr.14140) 379 : cluster [DBG] pgmap v373: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:03 smithi153 bash[35780]: cluster 2024-02-10T09:28:01.890012+0000 mgr.y (mgr.14140) 379 : cluster [DBG] pgmap v373: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:03 smithi153 bash[39117]: cluster 2024-02-10T09:28:01.890012+0000 mgr.y (mgr.14140) 379 : cluster [DBG] pgmap v373: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:05.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:05 smithi153 bash[35780]: cluster 2024-02-10T09:28:03.890585+0000 mgr.y (mgr.14140) 380 : cluster [DBG] pgmap v374: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:05.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:05 smithi153 bash[39117]: cluster 2024-02-10T09:28:03.890585+0000 mgr.y (mgr.14140) 380 : cluster [DBG] pgmap v374: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:05.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:05 smithi153 bash[55108]: time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:06.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:05 smithi204 bash[35809]: cluster 2024-02-10T09:28:03.890585+0000 mgr.y (mgr.14140) 380 : cluster [DBG] pgmap v374: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:06.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:06 smithi204 bash[44418]: time="2024-02-10T09:28:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:08.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:07 smithi204 bash[35809]: cluster 2024-02-10T09:28:05.891161+0000 mgr.y (mgr.14140) 381 : cluster [DBG] pgmap v375: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:07 smithi153 bash[35780]: cluster 2024-02-10T09:28:05.891161+0000 mgr.y (mgr.14140) 381 : cluster [DBG] pgmap v375: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:07 smithi153 bash[39117]: cluster 2024-02-10T09:28:05.891161+0000 mgr.y (mgr.14140) 381 : cluster [DBG] pgmap v375: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:10.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:09 smithi204 bash[35809]: cluster 2024-02-10T09:28:07.891602+0000 mgr.y (mgr.14140) 382 : cluster [DBG] pgmap v376: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:09 smithi153 bash[35780]: cluster 2024-02-10T09:28:07.891602+0000 mgr.y (mgr.14140) 382 : cluster [DBG] pgmap v376: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:09 smithi153 bash[39117]: cluster 2024-02-10T09:28:07.891602+0000 mgr.y (mgr.14140) 382 : cluster [DBG] pgmap v376: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:12.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:11 smithi204 bash[35809]: cluster 2024-02-10T09:28:09.892280+0000 mgr.y (mgr.14140) 383 : cluster [DBG] pgmap v377: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:11 smithi153 bash[35780]: cluster 2024-02-10T09:28:09.892280+0000 mgr.y (mgr.14140) 383 : cluster [DBG] pgmap v377: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:11 smithi153 bash[39117]: cluster 2024-02-10T09:28:09.892280+0000 mgr.y (mgr.14140) 383 : cluster [DBG] pgmap v377: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:13 smithi204 bash[35809]: cluster 2024-02-10T09:28:11.892978+0000 mgr.y (mgr.14140) 384 : cluster [DBG] pgmap v378: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:14.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:13 smithi153 bash[35780]: cluster 2024-02-10T09:28:11.892978+0000 mgr.y (mgr.14140) 384 : cluster [DBG] pgmap v378: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:13 smithi153 bash[39117]: cluster 2024-02-10T09:28:11.892978+0000 mgr.y (mgr.14140) 384 : cluster [DBG] pgmap v378: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:16.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:15 smithi204 bash[35809]: cluster 2024-02-10T09:28:13.893521+0000 mgr.y (mgr.14140) 385 : cluster [DBG] pgmap v379: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:15 smithi153 bash[35780]: cluster 2024-02-10T09:28:13.893521+0000 mgr.y (mgr.14140) 385 : cluster [DBG] pgmap v379: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:15 smithi153 bash[39117]: cluster 2024-02-10T09:28:13.893521+0000 mgr.y (mgr.14140) 385 : cluster [DBG] pgmap v379: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:18.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:17 smithi204 bash[35809]: cluster 2024-02-10T09:28:15.894099+0000 mgr.y (mgr.14140) 386 : cluster [DBG] pgmap v380: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:17 smithi153 bash[35780]: cluster 2024-02-10T09:28:15.894099+0000 mgr.y (mgr.14140) 386 : cluster [DBG] pgmap v380: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:17 smithi153 bash[39117]: cluster 2024-02-10T09:28:15.894099+0000 mgr.y (mgr.14140) 386 : cluster [DBG] pgmap v380: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:20.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:19 smithi204 bash[35809]: cluster 2024-02-10T09:28:17.894682+0000 mgr.y (mgr.14140) 387 : cluster [DBG] pgmap v381: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:19 smithi153 bash[39117]: cluster 2024-02-10T09:28:17.894682+0000 mgr.y (mgr.14140) 387 : cluster [DBG] pgmap v381: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:19 smithi153 bash[35780]: cluster 2024-02-10T09:28:17.894682+0000 mgr.y (mgr.14140) 387 : cluster [DBG] pgmap v381: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:21 smithi204 bash[35809]: cluster 2024-02-10T09:28:19.895244+0000 mgr.y (mgr.14140) 388 : cluster [DBG] pgmap v382: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:22.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:21 smithi153 bash[35780]: cluster 2024-02-10T09:28:19.895244+0000 mgr.y (mgr.14140) 388 : cluster [DBG] pgmap v382: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:21 smithi153 bash[39117]: cluster 2024-02-10T09:28:19.895244+0000 mgr.y (mgr.14140) 388 : cluster [DBG] pgmap v382: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:23 smithi204 bash[35809]: cluster 2024-02-10T09:28:21.895848+0000 mgr.y (mgr.14140) 389 : cluster [DBG] pgmap v383: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:23 smithi153 bash[35780]: cluster 2024-02-10T09:28:21.895848+0000 mgr.y (mgr.14140) 389 : cluster [DBG] pgmap v383: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:23 smithi153 bash[39117]: cluster 2024-02-10T09:28:21.895848+0000 mgr.y (mgr.14140) 389 : cluster [DBG] pgmap v383: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:25 smithi204 bash[35809]: cluster 2024-02-10T09:28:23.896592+0000 mgr.y (mgr.14140) 390 : cluster [DBG] pgmap v384: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:25 smithi153 bash[35780]: cluster 2024-02-10T09:28:23.896592+0000 mgr.y (mgr.14140) 390 : cluster [DBG] pgmap v384: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:25 smithi153 bash[39117]: cluster 2024-02-10T09:28:23.896592+0000 mgr.y (mgr.14140) 390 : cluster [DBG] pgmap v384: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:27 smithi204 bash[35809]: cluster 2024-02-10T09:28:25.897133+0000 mgr.y (mgr.14140) 391 : cluster [DBG] pgmap v385: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:27 smithi153 bash[35780]: cluster 2024-02-10T09:28:25.897133+0000 mgr.y (mgr.14140) 391 : cluster [DBG] pgmap v385: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:27 smithi153 bash[39117]: cluster 2024-02-10T09:28:25.897133+0000 mgr.y (mgr.14140) 391 : cluster [DBG] pgmap v385: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:30.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:29 smithi204 bash[35809]: cluster 2024-02-10T09:28:27.897695+0000 mgr.y (mgr.14140) 392 : cluster [DBG] pgmap v386: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:29 smithi153 bash[35780]: cluster 2024-02-10T09:28:27.897695+0000 mgr.y (mgr.14140) 392 : cluster [DBG] pgmap v386: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:29 smithi153 bash[39117]: cluster 2024-02-10T09:28:27.897695+0000 mgr.y (mgr.14140) 392 : cluster [DBG] pgmap v386: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:31 smithi204 bash[35809]: cluster 2024-02-10T09:28:29.898247+0000 mgr.y (mgr.14140) 393 : cluster [DBG] pgmap v387: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:31 smithi153 bash[35780]: cluster 2024-02-10T09:28:29.898247+0000 mgr.y (mgr.14140) 393 : cluster [DBG] pgmap v387: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:31 smithi153 bash[39117]: cluster 2024-02-10T09:28:29.898247+0000 mgr.y (mgr.14140) 393 : cluster [DBG] pgmap v387: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:33 smithi204 bash[35809]: cluster 2024-02-10T09:28:31.898859+0000 mgr.y (mgr.14140) 394 : cluster [DBG] pgmap v388: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:33 smithi153 bash[35780]: cluster 2024-02-10T09:28:31.898859+0000 mgr.y (mgr.14140) 394 : cluster [DBG] pgmap v388: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:33 smithi153 bash[39117]: cluster 2024-02-10T09:28:31.898859+0000 mgr.y (mgr.14140) 394 : cluster [DBG] pgmap v388: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:36.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:35 smithi204 bash[35809]: cluster 2024-02-10T09:28:33.899432+0000 mgr.y (mgr.14140) 395 : cluster [DBG] pgmap v389: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:35 smithi153 bash[35780]: cluster 2024-02-10T09:28:33.899432+0000 mgr.y (mgr.14140) 395 : cluster [DBG] pgmap v389: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:35 smithi153 bash[39117]: cluster 2024-02-10T09:28:33.899432+0000 mgr.y (mgr.14140) 395 : cluster [DBG] pgmap v389: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:37 smithi204 bash[35809]: cluster 2024-02-10T09:28:35.899955+0000 mgr.y (mgr.14140) 396 : cluster [DBG] pgmap v390: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:37 smithi153 bash[35780]: cluster 2024-02-10T09:28:35.899955+0000 mgr.y (mgr.14140) 396 : cluster [DBG] pgmap v390: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:37 smithi153 bash[39117]: cluster 2024-02-10T09:28:35.899955+0000 mgr.y (mgr.14140) 396 : cluster [DBG] pgmap v390: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:39 smithi204 bash[35809]: cluster 2024-02-10T09:28:37.900571+0000 mgr.y (mgr.14140) 397 : cluster [DBG] pgmap v391: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:39 smithi153 bash[35780]: cluster 2024-02-10T09:28:37.900571+0000 mgr.y (mgr.14140) 397 : cluster [DBG] pgmap v391: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:39 smithi153 bash[39117]: cluster 2024-02-10T09:28:37.900571+0000 mgr.y (mgr.14140) 397 : cluster [DBG] pgmap v391: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:41 smithi204 bash[35809]: cluster 2024-02-10T09:28:39.901148+0000 mgr.y (mgr.14140) 398 : cluster [DBG] pgmap v392: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:42.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:41 smithi153 bash[35780]: cluster 2024-02-10T09:28:39.901148+0000 mgr.y (mgr.14140) 398 : cluster [DBG] pgmap v392: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:41 smithi153 bash[39117]: cluster 2024-02-10T09:28:39.901148+0000 mgr.y (mgr.14140) 398 : cluster [DBG] pgmap v392: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:43.326 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[55108]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:28:43.326 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[55108]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:28:43.326 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[55108]: Error: 3 errors occurred while pulling: 2024-02-10T09:28:43.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[55108]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:43.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[55108]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:28:43.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[55108]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:43.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=125/n/a 2024-02-10T09:28:43.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:43 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: debug 2024-02-10T09:28:43.345+0000 7f90e3a5e700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:43.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 4247, in 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 968, in _default_image 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 2508, in command_deploy 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 1974, in run 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 696, in call_throws 2024-02-10T09:28:43.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: code, '\n'.join(err))) 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:43.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: Traceback (most recent call last): 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 4247, in 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 968, in _default_image 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 2508, in command_deploy 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 1974, in run 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: File "", line 696, in call_throws 2024-02-10T09:28:43.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[36014]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:43.902 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[35809]: cluster 2024-02-10T09:28:41.901839+0000 mgr.y (mgr.14140) 399 : cluster [DBG] pgmap v393: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:43.902 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[35809]: audit 2024-02-10T09:28:43.525546+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICxTCCAa0CEEsa3PR4GkJEsfVzTUQmK2QwDQYJKoZIhvcNAQENBQAwITENMAsG\nA1UECgwEQ2VwaDEQMA4GA1UEAwwHY2VwaGFkbTAeFw0yNDAyMTAwOTI4NDNaFw0z\nNDAyMDcwOTI4NDNaMCExDTALBgNVBAoMBENlcGgxEDAOBgNVBAMMB2NlcGhhZG0w\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAVJxZ2fBnvwJZqGmfrI/P\ndz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLGCfskeZUA\nS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3ZBLltRRe\nO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2pYRm8fVq\nussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBwotixXZ0V\nW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6WlVYHXEEH\nAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAAo85sR+akoPyl2AIYTbUtOoCfF0xII8\nyEfhJPF7QncYI/jQmP86VBNtLL1b6QCjtWGWlhhHvjQgECo0GNCL9qc2WYxFSpDF\nuNiWelpkZMZAddreMkp9gPmYcMz86c3IcqYNlj1qLu2IwrF6KcWq+ETd1uC5Sbhu\nNg+p85AmO2UaS+2hDDpUvlEm/nJNiomA3ayj7moOafYs9d4yyqKMKh8saNgRT8pZ\nA7boCoBWA7Q2fhc8EUGJK0OvMNbJhDIzQ8w8JOGOtHnc1EMejaoVCRYRP3zpue/X\n8xrMcb2pGAswWE0CMEPh165//O99+ymhmAGh6No3chwHJ0nGn2yV8vI=\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:28:43.902 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[35809]: audit 2024-02-10T09:28:43.528374+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICxTCCAa0CEEsa3PR4GkJEsfVzTUQmK2QwDQYJKoZIhvcNAQENBQAwITENMAsG\nA1UECgwEQ2VwaDEQMA4GA1UEAwwHY2VwaGFkbTAeFw0yNDAyMTAwOTI4NDNaFw0z\nNDAyMDcwOTI4NDNaMCExDTALBgNVBAoMBENlcGgxEDAOBgNVBAMMB2NlcGhhZG0w\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAVJxZ2fBnvwJZqGmfrI/P\ndz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLGCfskeZUA\nS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3ZBLltRRe\nO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2pYRm8fVq\nussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBwotixXZ0V\nW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6WlVYHXEEH\nAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAAo85sR+akoPyl2AIYTbUtOoCfF0xII8\nyEfhJPF7QncYI/jQmP86VBNtLL1b6QCjtWGWlhhHvjQgECo0GNCL9qc2WYxFSpDF\nuNiWelpkZMZAddreMkp9gPmYcMz86c3IcqYNlj1qLu2IwrF6KcWq+ETd1uC5Sbhu\nNg+p85AmO2UaS+2hDDpUvlEm/nJNiomA3ayj7moOafYs9d4yyqKMKh8saNgRT8pZ\nA7boCoBWA7Q2fhc8EUGJK0OvMNbJhDIzQ8w8JOGOtHnc1EMejaoVCRYRP3zpue/X\n8xrMcb2pGAswWE0CMEPh165//O99+ymhmAGh6No3chwHJ0nGn2yV8vI=\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:28:43.902 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[44418]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:28:43.902 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[44418]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:28:43.902 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[44418]: Error: 3 errors occurred while pulling: 2024-02-10T09:28:43.902 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[35809]: audit 2024-02-10T09:28:43.528779+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDAVJxZ2fBnvwJZ\nqGmfrI/Pdz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLG\nCfskeZUAS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3\nZBLltRReO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2\npYRm8fVqussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBw\notixXZ0VW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6W\nlVYHXEEHAgMBAAECggEAZzaA2W8tpl9JAZhxZnESlAGoIYvAUvFwK/xz94i9AgNX\ngox8nJMHKE1cNY06w7/ntUeBc086l3MjnseIcSCWJ5xR1KOECl5YodcrELbBCIx2\nvpTJ5Q+ev5DObmT6DJw3Dj7HnNxu9B2IdrhLs94jWha58KC8Jwe32YxstuKnKeJU\njZ7NzTNByePCMBd7ie9xAXMX6xBqtSUwdyYGzbDvznYN3vFZ/PGRQ4MuTSEHotzn\nav4OcJItRiOUgspnyRbkrzlQyT4nQxnM/8raaSyC2hxkrfpSUFhX/XOdPP710e11\nWFS9L4JAhxv8A+cpdBFoYRbhKUl5BHmhX3BS2foVkQKBgQDjGtpt59e/UWr98Et3\ngAarxKObIbBDlP9xJxMIcxN/9mEahnpmnEk2bR1E8sgH1BMxz4Kev1+YqYSFdYqb\n50qkpIjLsh3hEaBY9xfYDhRaOVnd4jQIM/6GKzILHqMhDcghiG7BEAgCdpHJC8K3\nSzsvzMmxpRAHcXTik9D/G6KIFQKBgQDYzRqLL/XCe7R/Ve8k9exTk41lYo9JXGNL\nVAQb0IY95/SRfCvHDg7cje13TXj33ZOhNM64ISTl17Q6zRbXSXgcQJ7vhzULhQf9\n3EaFVSIDUwnSHghSfDYk88ramX3Kvrb9b/J6FFlldk9xCXjjSW2tKmKc4qY/8J0g\nTKZ6r/GvqwKBgQCACgK+VikAF1IgC1sw64BXVUz8SHkNETMXQ3VqnR6QVYMXplva\nACESy1/mII8AQuPzgE/fWnv2IRSy/bsA/NzLiGwLk8devkZOAjlyOTXmTFFLVJ4L\nRe0pDa9wL3y/RPrhVP9EX9p0MQ8uby//kS20XFwWiL5JxACnQbliKt1upQKBgQC/\nrE3cYrF/mUjSD1KXtciTd29aBFI0BLdQwN3FDY2xq6JBAavel6IyAlTA1kCtuzsr\nGfntNnhdsGAKYIniGnVTXRpnIH2YSqvvF9FNTh/9GmwjVSP8cQ6ZiuR8cZmSJxy3\nYkLZBaanmnrPsBUttRx/LWrz0zoya3MvPDUvYOBncwKBgQCHHBd/pXIOxwo5hipr\nTbWKh8Q0SbwfCcym6Kjvxd4N04acwLax1p3UYpC2s6vlWleEf0/SMTpqdKT+5VoZ\nRVuNyoPLLMOOCTjxWdWV6NBfE659DyRdJG/KlZXqKhtEot7Eutt/VrSlA2x2oOrX\n6S1AEwJnJ011sSCo7/DBROobiQ==\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:28:43.903 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[35809]: audit 2024-02-10T09:28:43.530673+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDAVJxZ2fBnvwJZ\nqGmfrI/Pdz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLG\nCfskeZUAS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3\nZBLltRReO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2\npYRm8fVqussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBw\notixXZ0VW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6W\nlVYHXEEHAgMBAAECggEAZzaA2W8tpl9JAZhxZnESlAGoIYvAUvFwK/xz94i9AgNX\ngox8nJMHKE1cNY06w7/ntUeBc086l3MjnseIcSCWJ5xR1KOECl5YodcrELbBCIx2\nvpTJ5Q+ev5DObmT6DJw3Dj7HnNxu9B2IdrhLs94jWha58KC8Jwe32YxstuKnKeJU\njZ7NzTNByePCMBd7ie9xAXMX6xBqtSUwdyYGzbDvznYN3vFZ/PGRQ4MuTSEHotzn\nav4OcJItRiOUgspnyRbkrzlQyT4nQxnM/8raaSyC2hxkrfpSUFhX/XOdPP710e11\nWFS9L4JAhxv8A+cpdBFoYRbhKUl5BHmhX3BS2foVkQKBgQDjGtpt59e/UWr98Et3\ngAarxKObIbBDlP9xJxMIcxN/9mEahnpmnEk2bR1E8sgH1BMxz4Kev1+YqYSFdYqb\n50qkpIjLsh3hEaBY9xfYDhRaOVnd4jQIM/6GKzILHqMhDcghiG7BEAgCdpHJC8K3\nSzsvzMmxpRAHcXTik9D/G6KIFQKBgQDYzRqLL/XCe7R/Ve8k9exTk41lYo9JXGNL\nVAQb0IY95/SRfCvHDg7cje13TXj33ZOhNM64ISTl17Q6zRbXSXgcQJ7vhzULhQf9\n3EaFVSIDUwnSHghSfDYk88ramX3Kvrb9b/J6FFlldk9xCXjjSW2tKmKc4qY/8J0g\nTKZ6r/GvqwKBgQCACgK+VikAF1IgC1sw64BXVUz8SHkNETMXQ3VqnR6QVYMXplva\nACESy1/mII8AQuPzgE/fWnv2IRSy/bsA/NzLiGwLk8devkZOAjlyOTXmTFFLVJ4L\nRe0pDa9wL3y/RPrhVP9EX9p0MQ8uby//kS20XFwWiL5JxACnQbliKt1upQKBgQC/\nrE3cYrF/mUjSD1KXtciTd29aBFI0BLdQwN3FDY2xq6JBAavel6IyAlTA1kCtuzsr\nGfntNnhdsGAKYIniGnVTXRpnIH2YSqvvF9FNTh/9GmwjVSP8cQ6ZiuR8cZmSJxy3\nYkLZBaanmnrPsBUttRx/LWrz0zoya3MvPDUvYOBncwKBgQCHHBd/pXIOxwo5hipr\nTbWKh8Q0SbwfCcym6Kjvxd4N04acwLax1p3UYpC2s6vlWleEf0/SMTpqdKT+5VoZ\nRVuNyoPLLMOOCTjxWdWV6NBfE659DyRdJG/KlZXqKhtEot7Eutt/VrSlA2x2oOrX\n6S1AEwJnJ011sSCo7/DBROobiQ==\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:28:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[39117]: cluster 2024-02-10T09:28:41.901839+0000 mgr.y (mgr.14140) 399 : cluster [DBG] pgmap v393: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[39117]: audit 2024-02-10T09:28:43.525546+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICxTCCAa0CEEsa3PR4GkJEsfVzTUQmK2QwDQYJKoZIhvcNAQENBQAwITENMAsG\nA1UECgwEQ2VwaDEQMA4GA1UEAwwHY2VwaGFkbTAeFw0yNDAyMTAwOTI4NDNaFw0z\nNDAyMDcwOTI4NDNaMCExDTALBgNVBAoMBENlcGgxEDAOBgNVBAMMB2NlcGhhZG0w\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAVJxZ2fBnvwJZqGmfrI/P\ndz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLGCfskeZUA\nS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3ZBLltRRe\nO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2pYRm8fVq\nussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBwotixXZ0V\nW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6WlVYHXEEH\nAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAAo85sR+akoPyl2AIYTbUtOoCfF0xII8\nyEfhJPF7QncYI/jQmP86VBNtLL1b6QCjtWGWlhhHvjQgECo0GNCL9qc2WYxFSpDF\nuNiWelpkZMZAddreMkp9gPmYcMz86c3IcqYNlj1qLu2IwrF6KcWq+ETd1uC5Sbhu\nNg+p85AmO2UaS+2hDDpUvlEm/nJNiomA3ayj7moOafYs9d4yyqKMKh8saNgRT8pZ\nA7boCoBWA7Q2fhc8EUGJK0OvMNbJhDIzQ8w8JOGOtHnc1EMejaoVCRYRP3zpue/X\n8xrMcb2pGAswWE0CMEPh165//O99+ymhmAGh6No3chwHJ0nGn2yV8vI=\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:28:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[39117]: audit 2024-02-10T09:28:43.528374+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICxTCCAa0CEEsa3PR4GkJEsfVzTUQmK2QwDQYJKoZIhvcNAQENBQAwITENMAsG\nA1UECgwEQ2VwaDEQMA4GA1UEAwwHY2VwaGFkbTAeFw0yNDAyMTAwOTI4NDNaFw0z\nNDAyMDcwOTI4NDNaMCExDTALBgNVBAoMBENlcGgxEDAOBgNVBAMMB2NlcGhhZG0w\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAVJxZ2fBnvwJZqGmfrI/P\ndz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLGCfskeZUA\nS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3ZBLltRRe\nO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2pYRm8fVq\nussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBwotixXZ0V\nW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6WlVYHXEEH\nAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAAo85sR+akoPyl2AIYTbUtOoCfF0xII8\nyEfhJPF7QncYI/jQmP86VBNtLL1b6QCjtWGWlhhHvjQgECo0GNCL9qc2WYxFSpDF\nuNiWelpkZMZAddreMkp9gPmYcMz86c3IcqYNlj1qLu2IwrF6KcWq+ETd1uC5Sbhu\nNg+p85AmO2UaS+2hDDpUvlEm/nJNiomA3ayj7moOafYs9d4yyqKMKh8saNgRT8pZ\nA7boCoBWA7Q2fhc8EUGJK0OvMNbJhDIzQ8w8JOGOtHnc1EMejaoVCRYRP3zpue/X\n8xrMcb2pGAswWE0CMEPh165//O99+ymhmAGh6No3chwHJ0nGn2yV8vI=\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[35780]: cluster 2024-02-10T09:28:41.901839+0000 mgr.y (mgr.14140) 399 : cluster [DBG] pgmap v393: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[35780]: audit 2024-02-10T09:28:43.525546+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICxTCCAa0CEEsa3PR4GkJEsfVzTUQmK2QwDQYJKoZIhvcNAQENBQAwITENMAsG\nA1UECgwEQ2VwaDEQMA4GA1UEAwwHY2VwaGFkbTAeFw0yNDAyMTAwOTI4NDNaFw0z\nNDAyMDcwOTI4NDNaMCExDTALBgNVBAoMBENlcGgxEDAOBgNVBAMMB2NlcGhhZG0w\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAVJxZ2fBnvwJZqGmfrI/P\ndz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLGCfskeZUA\nS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3ZBLltRRe\nO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2pYRm8fVq\nussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBwotixXZ0V\nW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6WlVYHXEEH\nAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAAo85sR+akoPyl2AIYTbUtOoCfF0xII8\nyEfhJPF7QncYI/jQmP86VBNtLL1b6QCjtWGWlhhHvjQgECo0GNCL9qc2WYxFSpDF\nuNiWelpkZMZAddreMkp9gPmYcMz86c3IcqYNlj1qLu2IwrF6KcWq+ETd1uC5Sbhu\nNg+p85AmO2UaS+2hDDpUvlEm/nJNiomA3ayj7moOafYs9d4yyqKMKh8saNgRT8pZ\nA7boCoBWA7Q2fhc8EUGJK0OvMNbJhDIzQ8w8JOGOtHnc1EMejaoVCRYRP3zpue/X\n8xrMcb2pGAswWE0CMEPh165//O99+ymhmAGh6No3chwHJ0nGn2yV8vI=\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[35780]: audit 2024-02-10T09:28:43.528374+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICxTCCAa0CEEsa3PR4GkJEsfVzTUQmK2QwDQYJKoZIhvcNAQENBQAwITENMAsG\nA1UECgwEQ2VwaDEQMA4GA1UEAwwHY2VwaGFkbTAeFw0yNDAyMTAwOTI4NDNaFw0z\nNDAyMDcwOTI4NDNaMCExDTALBgNVBAoMBENlcGgxEDAOBgNVBAMMB2NlcGhhZG0w\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAVJxZ2fBnvwJZqGmfrI/P\ndz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLGCfskeZUA\nS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3ZBLltRRe\nO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2pYRm8fVq\nussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBwotixXZ0V\nW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6WlVYHXEEH\nAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAAo85sR+akoPyl2AIYTbUtOoCfF0xII8\nyEfhJPF7QncYI/jQmP86VBNtLL1b6QCjtWGWlhhHvjQgECo0GNCL9qc2WYxFSpDF\nuNiWelpkZMZAddreMkp9gPmYcMz86c3IcqYNlj1qLu2IwrF6KcWq+ETd1uC5Sbhu\nNg+p85AmO2UaS+2hDDpUvlEm/nJNiomA3ayj7moOafYs9d4yyqKMKh8saNgRT8pZ\nA7boCoBWA7Q2fhc8EUGJK0OvMNbJhDIzQ8w8JOGOtHnc1EMejaoVCRYRP3zpue/X\n8xrMcb2pGAswWE0CMEPh165//O99+ymhmAGh6No3chwHJ0nGn2yV8vI=\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[39117]: audit 2024-02-10T09:28:43.528779+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDAVJxZ2fBnvwJZ\nqGmfrI/Pdz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLG\nCfskeZUAS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3\nZBLltRReO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2\npYRm8fVqussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBw\notixXZ0VW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6W\nlVYHXEEHAgMBAAECggEAZzaA2W8tpl9JAZhxZnESlAGoIYvAUvFwK/xz94i9AgNX\ngox8nJMHKE1cNY06w7/ntUeBc086l3MjnseIcSCWJ5xR1KOECl5YodcrELbBCIx2\nvpTJ5Q+ev5DObmT6DJw3Dj7HnNxu9B2IdrhLs94jWha58KC8Jwe32YxstuKnKeJU\njZ7NzTNByePCMBd7ie9xAXMX6xBqtSUwdyYGzbDvznYN3vFZ/PGRQ4MuTSEHotzn\nav4OcJItRiOUgspnyRbkrzlQyT4nQxnM/8raaSyC2hxkrfpSUFhX/XOdPP710e11\nWFS9L4JAhxv8A+cpdBFoYRbhKUl5BHmhX3BS2foVkQKBgQDjGtpt59e/UWr98Et3\ngAarxKObIbBDlP9xJxMIcxN/9mEahnpmnEk2bR1E8sgH1BMxz4Kev1+YqYSFdYqb\n50qkpIjLsh3hEaBY9xfYDhRaOVnd4jQIM/6GKzILHqMhDcghiG7BEAgCdpHJC8K3\nSzsvzMmxpRAHcXTik9D/G6KIFQKBgQDYzRqLL/XCe7R/Ve8k9exTk41lYo9JXGNL\nVAQb0IY95/SRfCvHDg7cje13TXj33ZOhNM64ISTl17Q6zRbXSXgcQJ7vhzULhQf9\n3EaFVSIDUwnSHghSfDYk88ramX3Kvrb9b/J6FFlldk9xCXjjSW2tKmKc4qY/8J0g\nTKZ6r/GvqwKBgQCACgK+VikAF1IgC1sw64BXVUz8SHkNETMXQ3VqnR6QVYMXplva\nACESy1/mII8AQuPzgE/fWnv2IRSy/bsA/NzLiGwLk8devkZOAjlyOTXmTFFLVJ4L\nRe0pDa9wL3y/RPrhVP9EX9p0MQ8uby//kS20XFwWiL5JxACnQbliKt1upQKBgQC/\nrE3cYrF/mUjSD1KXtciTd29aBFI0BLdQwN3FDY2xq6JBAavel6IyAlTA1kCtuzsr\nGfntNnhdsGAKYIniGnVTXRpnIH2YSqvvF9FNTh/9GmwjVSP8cQ6ZiuR8cZmSJxy3\nYkLZBaanmnrPsBUttRx/LWrz0zoya3MvPDUvYOBncwKBgQCHHBd/pXIOxwo5hipr\nTbWKh8Q0SbwfCcym6Kjvxd4N04acwLax1p3UYpC2s6vlWleEf0/SMTpqdKT+5VoZ\nRVuNyoPLLMOOCTjxWdWV6NBfE659DyRdJG/KlZXqKhtEot7Eutt/VrSlA2x2oOrX\n6S1AEwJnJ011sSCo7/DBROobiQ==\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[39117]: audit 2024-02-10T09:28:43.530673+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDAVJxZ2fBnvwJZ\nqGmfrI/Pdz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLG\nCfskeZUAS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3\nZBLltRReO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2\npYRm8fVqussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBw\notixXZ0VW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6W\nlVYHXEEHAgMBAAECggEAZzaA2W8tpl9JAZhxZnESlAGoIYvAUvFwK/xz94i9AgNX\ngox8nJMHKE1cNY06w7/ntUeBc086l3MjnseIcSCWJ5xR1KOECl5YodcrELbBCIx2\nvpTJ5Q+ev5DObmT6DJw3Dj7HnNxu9B2IdrhLs94jWha58KC8Jwe32YxstuKnKeJU\njZ7NzTNByePCMBd7ie9xAXMX6xBqtSUwdyYGzbDvznYN3vFZ/PGRQ4MuTSEHotzn\nav4OcJItRiOUgspnyRbkrzlQyT4nQxnM/8raaSyC2hxkrfpSUFhX/XOdPP710e11\nWFS9L4JAhxv8A+cpdBFoYRbhKUl5BHmhX3BS2foVkQKBgQDjGtpt59e/UWr98Et3\ngAarxKObIbBDlP9xJxMIcxN/9mEahnpmnEk2bR1E8sgH1BMxz4Kev1+YqYSFdYqb\n50qkpIjLsh3hEaBY9xfYDhRaOVnd4jQIM/6GKzILHqMhDcghiG7BEAgCdpHJC8K3\nSzsvzMmxpRAHcXTik9D/G6KIFQKBgQDYzRqLL/XCe7R/Ve8k9exTk41lYo9JXGNL\nVAQb0IY95/SRfCvHDg7cje13TXj33ZOhNM64ISTl17Q6zRbXSXgcQJ7vhzULhQf9\n3EaFVSIDUwnSHghSfDYk88ramX3Kvrb9b/J6FFlldk9xCXjjSW2tKmKc4qY/8J0g\nTKZ6r/GvqwKBgQCACgK+VikAF1IgC1sw64BXVUz8SHkNETMXQ3VqnR6QVYMXplva\nACESy1/mII8AQuPzgE/fWnv2IRSy/bsA/NzLiGwLk8devkZOAjlyOTXmTFFLVJ4L\nRe0pDa9wL3y/RPrhVP9EX9p0MQ8uby//kS20XFwWiL5JxACnQbliKt1upQKBgQC/\nrE3cYrF/mUjSD1KXtciTd29aBFI0BLdQwN3FDY2xq6JBAavel6IyAlTA1kCtuzsr\nGfntNnhdsGAKYIniGnVTXRpnIH2YSqvvF9FNTh/9GmwjVSP8cQ6ZiuR8cZmSJxy3\nYkLZBaanmnrPsBUttRx/LWrz0zoya3MvPDUvYOBncwKBgQCHHBd/pXIOxwo5hipr\nTbWKh8Q0SbwfCcym6Kjvxd4N04acwLax1p3UYpC2s6vlWleEf0/SMTpqdKT+5VoZ\nRVuNyoPLLMOOCTjxWdWV6NBfE659DyRdJG/KlZXqKhtEot7Eutt/VrSlA2x2oOrX\n6S1AEwJnJ011sSCo7/DBROobiQ==\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[35780]: audit 2024-02-10T09:28:43.528779+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDAVJxZ2fBnvwJZ\nqGmfrI/Pdz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLG\nCfskeZUAS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3\nZBLltRReO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2\npYRm8fVqussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBw\notixXZ0VW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6W\nlVYHXEEHAgMBAAECggEAZzaA2W8tpl9JAZhxZnESlAGoIYvAUvFwK/xz94i9AgNX\ngox8nJMHKE1cNY06w7/ntUeBc086l3MjnseIcSCWJ5xR1KOECl5YodcrELbBCIx2\nvpTJ5Q+ev5DObmT6DJw3Dj7HnNxu9B2IdrhLs94jWha58KC8Jwe32YxstuKnKeJU\njZ7NzTNByePCMBd7ie9xAXMX6xBqtSUwdyYGzbDvznYN3vFZ/PGRQ4MuTSEHotzn\nav4OcJItRiOUgspnyRbkrzlQyT4nQxnM/8raaSyC2hxkrfpSUFhX/XOdPP710e11\nWFS9L4JAhxv8A+cpdBFoYRbhKUl5BHmhX3BS2foVkQKBgQDjGtpt59e/UWr98Et3\ngAarxKObIbBDlP9xJxMIcxN/9mEahnpmnEk2bR1E8sgH1BMxz4Kev1+YqYSFdYqb\n50qkpIjLsh3hEaBY9xfYDhRaOVnd4jQIM/6GKzILHqMhDcghiG7BEAgCdpHJC8K3\nSzsvzMmxpRAHcXTik9D/G6KIFQKBgQDYzRqLL/XCe7R/Ve8k9exTk41lYo9JXGNL\nVAQb0IY95/SRfCvHDg7cje13TXj33ZOhNM64ISTl17Q6zRbXSXgcQJ7vhzULhQf9\n3EaFVSIDUwnSHghSfDYk88ramX3Kvrb9b/J6FFlldk9xCXjjSW2tKmKc4qY/8J0g\nTKZ6r/GvqwKBgQCACgK+VikAF1IgC1sw64BXVUz8SHkNETMXQ3VqnR6QVYMXplva\nACESy1/mII8AQuPzgE/fWnv2IRSy/bsA/NzLiGwLk8devkZOAjlyOTXmTFFLVJ4L\nRe0pDa9wL3y/RPrhVP9EX9p0MQ8uby//kS20XFwWiL5JxACnQbliKt1upQKBgQC/\nrE3cYrF/mUjSD1KXtciTd29aBFI0BLdQwN3FDY2xq6JBAavel6IyAlTA1kCtuzsr\nGfntNnhdsGAKYIniGnVTXRpnIH2YSqvvF9FNTh/9GmwjVSP8cQ6ZiuR8cZmSJxy3\nYkLZBaanmnrPsBUttRx/LWrz0zoya3MvPDUvYOBncwKBgQCHHBd/pXIOxwo5hipr\nTbWKh8Q0SbwfCcym6Kjvxd4N04acwLax1p3UYpC2s6vlWleEf0/SMTpqdKT+5VoZ\nRVuNyoPLLMOOCTjxWdWV6NBfE659DyRdJG/KlZXqKhtEot7Eutt/VrSlA2x2oOrX\n6S1AEwJnJ011sSCo7/DBROobiQ==\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:28:44.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:43 smithi153 bash[35780]: audit 2024-02-10T09:28:43.530673+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDAVJxZ2fBnvwJZ\nqGmfrI/Pdz6UFOsv8613oGI1DdHBUP8Yu9k5v5gUw2Y7YJ9x4GBaQMXy/X7NzXLG\nCfskeZUAS2uTNE+0FWSNcVbslBhk1rG3Jijc4gYjMpplFeI55ga+XzjRFJGkDlu3\nZBLltRReO83MvyaA+1GbJSdQ1c/Jzd4Wa8LOhrRtYyAwtcSbaem0FNP0xLWsMQD2\npYRm8fVqussRzFdgNlDjdiCZwDJIICqF/s6EogB3EQb/53LHd/uQba6Tw6KDPsBw\notixXZ0VW9xy4t6DvHsuNmfqJp6Axp+36w+0CIuHeTRgHMtyInfdYr2Msge4nx6W\nlVYHXEEHAgMBAAECggEAZzaA2W8tpl9JAZhxZnESlAGoIYvAUvFwK/xz94i9AgNX\ngox8nJMHKE1cNY06w7/ntUeBc086l3MjnseIcSCWJ5xR1KOECl5YodcrELbBCIx2\nvpTJ5Q+ev5DObmT6DJw3Dj7HnNxu9B2IdrhLs94jWha58KC8Jwe32YxstuKnKeJU\njZ7NzTNByePCMBd7ie9xAXMX6xBqtSUwdyYGzbDvznYN3vFZ/PGRQ4MuTSEHotzn\nav4OcJItRiOUgspnyRbkrzlQyT4nQxnM/8raaSyC2hxkrfpSUFhX/XOdPP710e11\nWFS9L4JAhxv8A+cpdBFoYRbhKUl5BHmhX3BS2foVkQKBgQDjGtpt59e/UWr98Et3\ngAarxKObIbBDlP9xJxMIcxN/9mEahnpmnEk2bR1E8sgH1BMxz4Kev1+YqYSFdYqb\n50qkpIjLsh3hEaBY9xfYDhRaOVnd4jQIM/6GKzILHqMhDcghiG7BEAgCdpHJC8K3\nSzsvzMmxpRAHcXTik9D/G6KIFQKBgQDYzRqLL/XCe7R/Ve8k9exTk41lYo9JXGNL\nVAQb0IY95/SRfCvHDg7cje13TXj33ZOhNM64ISTl17Q6zRbXSXgcQJ7vhzULhQf9\n3EaFVSIDUwnSHghSfDYk88ramX3Kvrb9b/J6FFlldk9xCXjjSW2tKmKc4qY/8J0g\nTKZ6r/GvqwKBgQCACgK+VikAF1IgC1sw64BXVUz8SHkNETMXQ3VqnR6QVYMXplva\nACESy1/mII8AQuPzgE/fWnv2IRSy/bsA/NzLiGwLk8devkZOAjlyOTXmTFFLVJ4L\nRe0pDa9wL3y/RPrhVP9EX9p0MQ8uby//kS20XFwWiL5JxACnQbliKt1upQKBgQC/\nrE3cYrF/mUjSD1KXtciTd29aBFI0BLdQwN3FDY2xq6JBAavel6IyAlTA1kCtuzsr\nGfntNnhdsGAKYIniGnVTXRpnIH2YSqvvF9FNTh/9GmwjVSP8cQ6ZiuR8cZmSJxy3\nYkLZBaanmnrPsBUttRx/LWrz0zoya3MvPDUvYOBncwKBgQCHHBd/pXIOxwo5hipr\nTbWKh8Q0SbwfCcym6Kjvxd4N04acwLax1p3UYpC2s6vlWleEf0/SMTpqdKT+5VoZ\nRVuNyoPLLMOOCTjxWdWV6NBfE659DyRdJG/KlZXqKhtEot7Eutt/VrSlA2x2oOrX\n6S1AEwJnJ011sSCo7/DBROobiQ==\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:28:44.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[44418]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:44.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[44418]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:28:44.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 bash[44418]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:44.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=125/n/a 2024-02-10T09:28:44.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: cephadm 2024-02-10T09:28:43.346523+0000 mgr.y (mgr.14140) 400 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: cephadm 2024-02-10T09:28:43.346523+0000 mgr.y (mgr.14140) 400 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:28:45.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: cephadm 2024-02-10T09:28:43.346637+0000 mgr.y (mgr.14140) 401 : cephadm [WRN] Failed to apply alertmanager spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi153', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 4247, in 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 968, in _default_image 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 1974, in run 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: File "", line 696, in call_throws 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[35780]: cephadm 2024-02-10T09:28:43.531111+0000 mgr.y (mgr.14140) 402 : cephadm [INF] Deploying daemon grafana.a on smithi204 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: cephadm 2024-02-10T09:28:43.346637+0000 mgr.y (mgr.14140) 401 : cephadm [WRN] Failed to apply alertmanager spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi153', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:28:45.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 4247, in 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 968, in _default_image 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 1974, in run 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: File "", line 696, in call_throws 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:44 smithi153 bash[39117]: cephadm 2024-02-10T09:28:43.531111+0000 mgr.y (mgr.14140) 402 : cephadm [INF] Deploying daemon grafana.a on smithi204 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: cephadm 2024-02-10T09:28:43.346523+0000 mgr.y (mgr.14140) 400 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1613, in _run_cephadm 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:28:45.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: cephadm 2024-02-10T09:28:43.346637+0000 mgr.y (mgr.14140) 401 : cephadm [WRN] Failed to apply alertmanager spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi153', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False}): cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon alertmanager.a ... 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9093 ... 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Verifying port 9094 ... 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Resolving "prom/alertmanager" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/alertmanager:latest... 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:26:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:27:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr time="2024-02-10T09:28:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/alertmanager:latest... 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Trying to pull registry.redhat.io/prom/alertmanager:latest... 2024-02-10T09:28:45.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr Error: 3 errors occurred while pulling: 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://prom/alertmanager:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:latest: reading manifest latest in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.access.redhat.com/prom/alertmanager:latest: reading manifest latest in registry.access.redhat.com/prom/alertmanager: name unknown: Repo not found 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: INFO:cephadm:stat:stderr * initializing source docker://registry.redhat.io/prom/alertmanager:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 4247, in 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 968, in _default_image 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 2508, in command_deploy 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 2454, in extract_uid_gid_monitoring 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 1535, in extract_uid_gid 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 1974, in run 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: File "", line 696, in call_throws 2024-02-10T09:28:45.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=prom/alertmanager -e NODE_NAME=smithi153 --entrypoint stat prom/alertmanager -c %u %g /etc/alertmanager 2024-02-10T09:28:45.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:44 smithi204 bash[35809]: cephadm 2024-02-10T09:28:43.531111+0000 mgr.y (mgr.14140) 402 : cephadm [INF] Deploying daemon grafana.a on smithi204 2024-02-10T09:28:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:45 smithi153 bash[35780]: cluster 2024-02-10T09:28:43.902686+0000 mgr.y (mgr.14140) 403 : cluster [DBG] pgmap v394: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:45 smithi153 bash[39117]: cluster 2024-02-10T09:28:43.902686+0000 mgr.y (mgr.14140) 403 : cluster [DBG] pgmap v394: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:45 smithi204 bash[35809]: cluster 2024-02-10T09:28:43.902686+0000 mgr.y (mgr.14140) 403 : cluster [DBG] pgmap v394: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:47 smithi153 bash[35780]: cluster 2024-02-10T09:28:45.903252+0000 mgr.y (mgr.14140) 404 : cluster [DBG] pgmap v395: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:47 smithi153 bash[39117]: cluster 2024-02-10T09:28:45.903252+0000 mgr.y (mgr.14140) 404 : cluster [DBG] pgmap v395: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:47 smithi204 bash[35809]: cluster 2024-02-10T09:28:45.903252+0000 mgr.y (mgr.14140) 404 : cluster [DBG] pgmap v395: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:49 smithi153 bash[35780]: cluster 2024-02-10T09:28:47.903891+0000 mgr.y (mgr.14140) 405 : cluster [DBG] pgmap v396: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:49 smithi153 bash[39117]: cluster 2024-02-10T09:28:47.903891+0000 mgr.y (mgr.14140) 405 : cluster [DBG] pgmap v396: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:49 smithi204 bash[35809]: cluster 2024-02-10T09:28:47.903891+0000 mgr.y (mgr.14140) 405 : cluster [DBG] pgmap v396: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:51 smithi153 bash[35780]: cluster 2024-02-10T09:28:49.904501+0000 mgr.y (mgr.14140) 406 : cluster [DBG] pgmap v397: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:51 smithi153 bash[39117]: cluster 2024-02-10T09:28:49.904501+0000 mgr.y (mgr.14140) 406 : cluster [DBG] pgmap v397: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:52.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:51 smithi204 bash[35809]: cluster 2024-02-10T09:28:49.904501+0000 mgr.y (mgr.14140) 406 : cluster [DBG] pgmap v397: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:53.856 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:28:53.856 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:28:53.857 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:28:53.857 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:28:53.857 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 podman[55947]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T09:28:53.857 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 systemd[1]: Started Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:28:53.857 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 bash[55964]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:53.857 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:28:53 smithi153 bash[55964]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:28:54.108 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:53 smithi204 bash[35809]: cluster 2024-02-10T09:28:51.905070+0000 mgr.y (mgr.14140) 407 : cluster [DBG] pgmap v398: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:54.108 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:28:54.108 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:28:54.108 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:28:54.108 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:28:54.108 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 podman[45084]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T09:28:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:53 smithi153 bash[35780]: cluster 2024-02-10T09:28:51.905070+0000 mgr.y (mgr.14140) 407 : cluster [DBG] pgmap v398: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:53 smithi153 bash[39117]: cluster 2024-02-10T09:28:51.905070+0000 mgr.y (mgr.14140) 407 : cluster [DBG] pgmap v398: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:54.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 systemd[1]: Started Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:28:54.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 bash[45101]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:28:54.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:28:54 smithi204 bash[45101]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:28:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:55 smithi153 bash[35780]: cluster 2024-02-10T09:28:53.905829+0000 mgr.y (mgr.14140) 408 : cluster [DBG] pgmap v399: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:55 smithi153 bash[39117]: cluster 2024-02-10T09:28:53.905829+0000 mgr.y (mgr.14140) 408 : cluster [DBG] pgmap v399: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:55 smithi204 bash[35809]: cluster 2024-02-10T09:28:53.905829+0000 mgr.y (mgr.14140) 408 : cluster [DBG] pgmap v399: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:58.163 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:57 smithi204 bash[35809]: cluster 2024-02-10T09:28:55.906407+0000 mgr.y (mgr.14140) 409 : cluster [DBG] pgmap v400: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:57 smithi153 bash[35780]: cluster 2024-02-10T09:28:55.906407+0000 mgr.y (mgr.14140) 409 : cluster [DBG] pgmap v400: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:28:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:57 smithi153 bash[39117]: cluster 2024-02-10T09:28:55.906407+0000 mgr.y (mgr.14140) 409 : cluster [DBG] pgmap v400: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:00.146 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:28:59 smithi204 bash[35809]: cluster 2024-02-10T09:28:57.906942+0000 mgr.y (mgr.14140) 410 : cluster [DBG] pgmap v401: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:28:59 smithi153 bash[35780]: cluster 2024-02-10T09:28:57.906942+0000 mgr.y (mgr.14140) 410 : cluster [DBG] pgmap v401: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:28:59 smithi153 bash[39117]: cluster 2024-02-10T09:28:57.906942+0000 mgr.y (mgr.14140) 410 : cluster [DBG] pgmap v401: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:00.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:00 smithi204 podman[45290]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T09:29:00.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:00 smithi204 systemd[1]: Started Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:29:01.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.467749+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:29:01.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.467749+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:29:01.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.471035+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.472048+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_zone", "value": "z"}]: dispatch 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.476462+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_zone", "value": "z"}]': finished 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.477339+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r", "name": "rgw_realm", "value": "r"}]: dispatch 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.481186+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r", "name": "rgw_realm", "value": "r"}]': finished 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.482595+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_frontends", "value": "beast port=8000"}]: dispatch 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.487957+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_frontends", "value": "beast port=8000"}]': finished 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.488864+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]: dispatch 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.493094+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]': finished 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.493869+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[39117]: audit 2024-02-10T09:29:00.494783+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.r.z.smithi153.zyveqt", "key": "container_image"}]: dispatch 2024-02-10T09:29:01.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.471035+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.472048+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_zone", "value": "z"}]: dispatch 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.476462+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_zone", "value": "z"}]': finished 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.477339+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r", "name": "rgw_realm", "value": "r"}]: dispatch 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.481186+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r", "name": "rgw_realm", "value": "r"}]': finished 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.482595+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_frontends", "value": "beast port=8000"}]: dispatch 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.487957+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_frontends", "value": "beast port=8000"}]': finished 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.488864+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]: dispatch 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.493094+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]': finished 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.493869+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:29:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:00 smithi153 bash[35780]: audit 2024-02-10T09:29:00.494783+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.r.z.smithi153.zyveqt", "key": "container_image"}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.467749+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.471035+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.472048+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_zone", "value": "z"}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.476462+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_zone", "value": "z"}]': finished 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.477339+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r", "name": "rgw_realm", "value": "r"}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.481186+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r", "name": "rgw_realm", "value": "r"}]': finished 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.482595+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_frontends", "value": "beast port=8000"}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.487957+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "config set", "who": "client.rgw.r.z", "name": "rgw_frontends", "value": "beast port=8000"}]': finished 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.488864+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.493094+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]': finished 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.493869+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:29:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:00 smithi204 bash[35809]: audit 2024-02-10T09:29:00.494783+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.r.z.smithi153.zyveqt", "key": "container_image"}]: dispatch 2024-02-10T09:29:01.879 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:01 smithi153 bash[35780]: cluster 2024-02-10T09:28:59.907487+0000 mgr.y (mgr.14140) 411 : cluster [DBG] pgmap v402: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:01.879 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:01 smithi153 bash[35780]: cephadm 2024-02-10T09:29:00.494389+0000 mgr.y (mgr.14140) 412 : cephadm [INF] Deploying daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:29:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:01 smithi153 bash[39117]: cluster 2024-02-10T09:28:59.907487+0000 mgr.y (mgr.14140) 411 : cluster [DBG] pgmap v402: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:01 smithi153 bash[39117]: cephadm 2024-02-10T09:29:00.494389+0000 mgr.y (mgr.14140) 412 : cephadm [INF] Deploying daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:29:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:01 smithi204 bash[35809]: cluster 2024-02-10T09:28:59.907487+0000 mgr.y (mgr.14140) 411 : cluster [DBG] pgmap v402: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:01 smithi204 bash[35809]: cephadm 2024-02-10T09:29:00.494389+0000 mgr.y (mgr.14140) 412 : cephadm [INF] Deploying daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Starting Grafana" logger=server version=6.6.2 commit=3fa63cfc34 branch=HEAD compiled=2020-02-20T12:03:49+0000 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T09:29:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Initializing SqlStore" logger=server 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Starting DB migration" logger=migrator 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-10T09:29:02.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-10T09:29:02.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T09:29:02.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-10T09:29:02.337 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T09:29:02.338 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-10T09:29:02.339 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T09:29:02.340 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-10T09:29:02.341 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-10T09:29:02.342 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-10T09:29:02.343 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-10T09:29:02.344 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-10T09:29:02.345 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-10T09:29:02.346 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-10T09:29:02.347 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing HTTPServer" logger=server 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing PluginManager" logger=server 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Starting plugin search" logger=plugins 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing HooksService" logger=server 2024-02-10T09:29:02.348 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing RemoteCache" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing QuotaService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing ServerLockService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing RenderingService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing AlertEngine" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing LoginService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing SearchService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing TracingService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing CleanUpService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing NotificationService" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-10T09:29:02.349 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2024-02-10T09:29:02.350 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin." logger=rendering renderer=phantomJS 2024-02-10T09:29:02.350 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="Initializing Stream Manager" 2024-02-10T09:29:02.350 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[45306]: t=2024-02-10T09:29:02+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3000 protocol=https subUrl= socket= 2024-02-10T09:29:03.141 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[35809]: audit 2024-02-10T09:29:02.666075+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:29:03.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[35809]: audit 2024-02-10T09:29:02.669098+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:29:03.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[35809]: audit 2024-02-10T09:29:02.669614+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]: dispatch 2024-02-10T09:29:03.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[35809]: audit 2024-02-10T09:29:02.675234+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]': finished 2024-02-10T09:29:03.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[35809]: audit 2024-02-10T09:29:02.675984+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:29:03.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:02 smithi204 bash[35809]: audit 2024-02-10T09:29:02.676622+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.r.z.smithi204.kmhqqg", "key": "container_image"}]: dispatch 2024-02-10T09:29:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[39117]: audit 2024-02-10T09:29:02.666075+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:29:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[39117]: audit 2024-02-10T09:29:02.669098+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:29:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[39117]: audit 2024-02-10T09:29:02.669614+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]: dispatch 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[39117]: audit 2024-02-10T09:29:02.675234+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]': finished 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[39117]: audit 2024-02-10T09:29:02.675984+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[39117]: audit 2024-02-10T09:29:02.676622+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.r.z.smithi204.kmhqqg", "key": "container_image"}]: dispatch 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[35780]: audit 2024-02-10T09:29:02.666075+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]: dispatch 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[35780]: audit 2024-02-10T09:29:02.669098+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:17:13.974966\"}"}]': finished 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[35780]: audit 2024-02-10T09:29:02.669614+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]: dispatch 2024-02-10T09:29:03.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[35780]: audit 2024-02-10T09:29:02.675234+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow rw", "mgr", "allow rw", "osd", "allow rwx"]}]': finished 2024-02-10T09:29:03.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[35780]: audit 2024-02-10T09:29:02.675984+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:29:03.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:02 smithi153 bash[35780]: audit 2024-02-10T09:29:02.676622+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.r.z.smithi204.kmhqqg", "key": "container_image"}]: dispatch 2024-02-10T09:29:03.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:03 smithi204 bash[35809]: cluster 2024-02-10T09:29:01.908056+0000 mgr.y (mgr.14140) 413 : cluster [DBG] pgmap v403: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:03.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:03 smithi204 bash[35809]: cephadm 2024-02-10T09:29:02.676399+0000 mgr.y (mgr.14140) 414 : cephadm [INF] Deploying daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:29:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:03 smithi153 bash[35780]: cluster 2024-02-10T09:29:01.908056+0000 mgr.y (mgr.14140) 413 : cluster [DBG] pgmap v403: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:03 smithi153 bash[35780]: cephadm 2024-02-10T09:29:02.676399+0000 mgr.y (mgr.14140) 414 : cephadm [INF] Deploying daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:29:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:03 smithi153 bash[39117]: cluster 2024-02-10T09:29:01.908056+0000 mgr.y (mgr.14140) 413 : cluster [DBG] pgmap v403: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:03 smithi153 bash[39117]: cephadm 2024-02-10T09:29:02.676399+0000 mgr.y (mgr.14140) 414 : cephadm [INF] Deploying daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:29:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[35780]: audit 2024-02-10T09:29:04.322084+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:29:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[35780]: audit 2024-02-10T09:29:04.325296+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:29:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[39117]: audit 2024-02-10T09:29:04.322084+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:29:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[39117]: audit 2024-02-10T09:29:04.325296+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:29:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[35780]: audit 2024-02-10T09:29:04.531485+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]: dispatch 2024-02-10T09:29:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[35780]: audit 2024-02-10T09:29:04.534156+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]': finished 2024-02-10T09:29:05.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[35780]: audit 2024-02-10T09:29:04.534709+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:29:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[39117]: audit 2024-02-10T09:29:04.531485+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]: dispatch 2024-02-10T09:29:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[39117]: audit 2024-02-10T09:29:04.534156+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]': finished 2024-02-10T09:29:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:04 smithi153 bash[39117]: audit 2024-02-10T09:29:04.534709+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:29:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:04 smithi204 bash[35809]: audit 2024-02-10T09:29:04.322084+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]: dispatch 2024-02-10T09:29:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:04 smithi204 bash[35809]: audit 2024-02-10T09:29:04.325296+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:17:49.580619\"}"}]': finished 2024-02-10T09:29:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:04 smithi204 bash[35809]: audit 2024-02-10T09:29:04.531485+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]: dispatch 2024-02-10T09:29:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:04 smithi204 bash[35809]: audit 2024-02-10T09:29:04.534156+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.096963\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097055\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097084\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097110\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097136\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097162\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097187\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:00.097213\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]': finished 2024-02-10T09:29:05.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:04 smithi204 bash[35809]: audit 2024-02-10T09:29:04.534709+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:29:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:05 smithi153 bash[35780]: cluster 2024-02-10T09:29:03.908726+0000 mgr.y (mgr.14140) 415 : cluster [DBG] pgmap v404: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:05 smithi153 bash[39117]: cluster 2024-02-10T09:29:03.908726+0000 mgr.y (mgr.14140) 415 : cluster [DBG] pgmap v404: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:05 smithi204 bash[35809]: cluster 2024-02-10T09:29:03.908726+0000 mgr.y (mgr.14140) 415 : cluster [DBG] pgmap v404: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[39117]: audit 2024-02-10T09:29:05.972685+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971589\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971681\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971710\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971763\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971792\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971819\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971846\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971874\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971890\", \"created\": \"2024-02-10T09:29:02.635166\", \"started\": \"2024-02-10T09:29:02.666766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_daemon_update\": \"2024-02-10T09:29:05.971966\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]: dispatch 2024-02-10T09:29:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[35780]: audit 2024-02-10T09:29:05.972685+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971589\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971681\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971710\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971763\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971792\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971819\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971846\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971874\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971890\", \"created\": \"2024-02-10T09:29:02.635166\", \"started\": \"2024-02-10T09:29:02.666766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_daemon_update\": \"2024-02-10T09:29:05.971966\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]: dispatch 2024-02-10T09:29:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[39117]: audit 2024-02-10T09:29:05.975604+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971589\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971681\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971710\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971763\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971792\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971819\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971846\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971874\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971890\", \"created\": \"2024-02-10T09:29:02.635166\", \"started\": \"2024-02-10T09:29:02.666766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_daemon_update\": \"2024-02-10T09:29:05.971966\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]': finished 2024-02-10T09:29:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[35780]: audit 2024-02-10T09:29:05.975604+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971589\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971681\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971710\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971763\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971792\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971819\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971846\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971874\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971890\", \"created\": \"2024-02-10T09:29:02.635166\", \"started\": \"2024-02-10T09:29:02.666766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_daemon_update\": \"2024-02-10T09:29:05.971966\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]': finished 2024-02-10T09:29:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[39117]: audit 2024-02-10T09:29:06.174939+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]: dispatch 2024-02-10T09:29:07.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[35780]: audit 2024-02-10T09:29:06.174939+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]: dispatch 2024-02-10T09:29:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[39117]: audit 2024-02-10T09:29:06.177596+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]': finished 2024-02-10T09:29:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[39117]: audit 2024-02-10T09:29:06.178148+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:29:07.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[35780]: audit 2024-02-10T09:29:06.177596+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]': finished 2024-02-10T09:29:07.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:06 smithi153 bash[35780]: audit 2024-02-10T09:29:06.178148+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:29:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:06 smithi204 bash[35809]: audit 2024-02-10T09:29:05.972685+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971589\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971681\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971710\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971763\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971792\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971819\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971846\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971874\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971890\", \"created\": \"2024-02-10T09:29:02.635166\", \"started\": \"2024-02-10T09:29:02.666766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_daemon_update\": \"2024-02-10T09:29:05.971966\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]: dispatch 2024-02-10T09:29:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:06 smithi204 bash[35809]: audit 2024-02-10T09:29:05.975604+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971589\", \"created\": \"2024-02-10T09:16:41.486607\", \"started\": \"2024-02-10T09:16:45.962801\"}, \"mgr.y\": {\"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971681\", \"created\": \"2024-02-10T09:16:47.620498\", \"started\": \"2024-02-10T09:16:47.651661\"}, \"mon.c\": {\"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971710\", \"created\": \"2024-02-10T09:18:01.850164\", \"started\": \"2024-02-10T09:18:01.883088\"}, \"osd.0\": {\"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971763\", \"created\": \"2024-02-10T09:18:52.144251\", \"started\": \"2024-02-10T09:19:13.914993\"}, \"osd.1\": {\"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971792\", \"created\": \"2024-02-10T09:19:12.921872\", \"started\": \"2024-02-10T09:19:35.395036\"}, \"osd.2\": {\"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971819\", \"created\": \"2024-02-10T09:19:36.993433\", \"started\": \"2024-02-10T09:19:39.009093\"}, \"osd.3\": {\"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971846\", \"created\": \"2024-02-10T09:19:55.310100\", \"started\": \"2024-02-10T09:19:57.924701\"}, \"node-exporter.a\": {\"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971874\", \"created\": \"2024-02-10T09:26:14.154248\"}, \"rgw.r.z.smithi153.zyveqt\": {\"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:05.971890\", \"created\": \"2024-02-10T09:29:02.635166\", \"started\": \"2024-02-10T09:29:02.666766\"}}, \"devices\": [{\"rejected_reasons\": [\"LVM detected\", \"locked\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650\"}}, \"last_daemon_update\": \"2024-02-10T09:29:05.971966\", \"last_device_update\": \"2024-02-10T09:20:01.571358\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.153\"]}, \"last_host_check\": \"2024-02-10T09:29:04.530747\"}"}]': finished 2024-02-10T09:29:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:06 smithi204 bash[35809]: audit 2024-02-10T09:29:06.174939+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]: dispatch 2024-02-10T09:29:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:06 smithi204 bash[35809]: audit 2024-02-10T09:29:06.177596+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178069\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178162\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178198\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178229\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178260\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178290\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:27:02.178320\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"status\": 1, \"status_desc\": \"starting\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"status\": 1, \"status_desc\": \"starting\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]': finished 2024-02-10T09:29:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:06 smithi204 bash[35809]: audit 2024-02-10T09:29:06.178148+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2024-02-10T09:29:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:07 smithi153 bash[39117]: cluster 2024-02-10T09:29:05.909263+0000 mgr.y (mgr.14140) 416 : cluster [DBG] pgmap v405: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 65 op/s 2024-02-10T09:29:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:07 smithi153 bash[35780]: cluster 2024-02-10T09:29:05.909263+0000 mgr.y (mgr.14140) 416 : cluster [DBG] pgmap v405: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 65 op/s 2024-02-10T09:29:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:07 smithi204 bash[35809]: cluster 2024-02-10T09:29:05.909263+0000 mgr.y (mgr.14140) 416 : cluster [DBG] pgmap v405: 129 pgs: 129 active+clean; 6.9 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 65 op/s 2024-02-10T09:29:09.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:08 smithi204 bash[35809]: cluster 2024-02-10T09:29:07.909689+0000 mgr.y (mgr.14140) 417 : cluster [DBG] pgmap v406: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 50 KiB/s rd, 170 B/s wr, 91 op/s 2024-02-10T09:29:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:08 smithi153 bash[39117]: cluster 2024-02-10T09:29:07.909689+0000 mgr.y (mgr.14140) 417 : cluster [DBG] pgmap v406: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 50 KiB/s rd, 170 B/s wr, 91 op/s 2024-02-10T09:29:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:08 smithi153 bash[35780]: cluster 2024-02-10T09:29:07.909689+0000 mgr.y (mgr.14140) 417 : cluster [DBG] pgmap v406: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 50 KiB/s rd, 170 B/s wr, 91 op/s 2024-02-10T09:29:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:09 smithi153 bash[35780]: audit 2024-02-10T09:29:09.008010+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006764\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006859\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006895\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006928\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006975\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007005\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007034\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007053\", \"created\": \"2024-02-10T09:29:00.434804\", \"started\": \"2024-02-10T09:29:00.545771\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007083\", \"created\": \"2024-02-10T09:29:04.291747\", \"started\": \"2024-02-10T09:29:04.321392\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_daemon_update\": \"2024-02-10T09:29:09.007183\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]: dispatch 2024-02-10T09:29:10.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:09 smithi153 bash[39117]: audit 2024-02-10T09:29:09.008010+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006764\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006859\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006895\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006928\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006975\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007005\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007034\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007053\", \"created\": \"2024-02-10T09:29:00.434804\", \"started\": \"2024-02-10T09:29:00.545771\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007083\", \"created\": \"2024-02-10T09:29:04.291747\", \"started\": \"2024-02-10T09:29:04.321392\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_daemon_update\": \"2024-02-10T09:29:09.007183\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]: dispatch 2024-02-10T09:29:10.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:09 smithi153 bash[35780]: audit 2024-02-10T09:29:09.011767+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006764\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006859\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006895\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006928\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006975\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007005\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007034\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007053\", \"created\": \"2024-02-10T09:29:00.434804\", \"started\": \"2024-02-10T09:29:00.545771\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007083\", \"created\": \"2024-02-10T09:29:04.291747\", \"started\": \"2024-02-10T09:29:04.321392\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_daemon_update\": \"2024-02-10T09:29:09.007183\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]': finished 2024-02-10T09:29:10.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:09 smithi153 bash[35780]: cephadm 2024-02-10T09:29:09.015955+0000 mgr.y (mgr.14140) 418 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:29:10.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:09 smithi153 bash[39117]: audit 2024-02-10T09:29:09.011767+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006764\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006859\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006895\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006928\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006975\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007005\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007034\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007053\", \"created\": \"2024-02-10T09:29:00.434804\", \"started\": \"2024-02-10T09:29:00.545771\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007083\", \"created\": \"2024-02-10T09:29:04.291747\", \"started\": \"2024-02-10T09:29:04.321392\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_daemon_update\": \"2024-02-10T09:29:09.007183\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]': finished 2024-02-10T09:29:10.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:09 smithi153 bash[39117]: cephadm 2024-02-10T09:29:09.015955+0000 mgr.y (mgr.14140) 418 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:29:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:09 smithi204 bash[35809]: audit 2024-02-10T09:29:09.008010+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006764\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006859\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006895\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006928\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006975\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007005\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007034\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007053\", \"created\": \"2024-02-10T09:29:00.434804\", \"started\": \"2024-02-10T09:29:00.545771\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007083\", \"created\": \"2024-02-10T09:29:04.291747\", \"started\": \"2024-02-10T09:29:04.321392\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_daemon_update\": \"2024-02-10T09:29:09.007183\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]: dispatch 2024-02-10T09:29:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:09 smithi204 bash[35809]: audit 2024-02-10T09:29:09.011767+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14140 172.21.15.153:0/1792798376' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"b\", \"daemon_type\": \"mon\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006764\", \"created\": \"2024-02-10T09:18:21.548104\", \"started\": \"2024-02-10T09:18:21.578355\"}, \"mgr.x\": {\"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"daemon_id\": \"x\", \"daemon_type\": \"mgr\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006859\", \"created\": \"2024-02-10T09:18:35.961903\", \"started\": \"2024-02-10T09:26:56.123455\"}, \"osd.4\": {\"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"4\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006895\", \"created\": \"2024-02-10T09:20:10.443566\", \"started\": \"2024-02-10T09:20:12.035540\"}, \"osd.5\": {\"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"5\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006928\", \"created\": \"2024-02-10T09:20:28.489307\", \"started\": \"2024-02-10T09:20:30.785766\"}, \"osd.6\": {\"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"6\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.006975\", \"created\": \"2024-02-10T09:20:47.002041\", \"started\": \"2024-02-10T09:20:49.479446\"}, \"osd.7\": {\"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"7\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007005\", \"created\": \"2024-02-10T09:21:05.721771\", \"started\": \"2024-02-10T09:21:07.713166\"}, \"node-exporter.b\": {\"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"daemon_id\": \"b\", \"daemon_type\": \"node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007034\", \"created\": \"2024-02-10T09:26:14.659269\"}, \"grafana.a\": {\"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"grafana\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007053\", \"created\": \"2024-02-10T09:29:00.434804\", \"started\": \"2024-02-10T09:29:00.545771\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_type\": \"rgw\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2024-02-10T09:29:09.007083\", \"created\": \"2024-02-10T09:29:04.291747\", \"started\": \"2024-02-10T09:29:04.321392\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"Insufficient space (<5GB) on vgs\", \"LVM detected\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\"}], \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775\"}}, \"last_daemon_update\": \"2024-02-10T09:29:09.007183\", \"last_device_update\": \"2024-02-10T09:21:11.911631\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.204\"]}, \"last_host_check\": \"2024-02-10T09:29:06.174259\"}"}]': finished 2024-02-10T09:29:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:09 smithi204 bash[35809]: cephadm 2024-02-10T09:29:09.015955+0000 mgr.y (mgr.14140) 418 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:29:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:11 smithi204 bash[35809]: cluster 2024-02-10T09:29:09.910281+0000 mgr.y (mgr.14140) 419 : cluster [DBG] pgmap v407: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 91 KiB/s rd, 170 B/s wr, 164 op/s 2024-02-10T09:29:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:11 smithi153 bash[35780]: cluster 2024-02-10T09:29:09.910281+0000 mgr.y (mgr.14140) 419 : cluster [DBG] pgmap v407: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 91 KiB/s rd, 170 B/s wr, 164 op/s 2024-02-10T09:29:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:11 smithi153 bash[39117]: cluster 2024-02-10T09:29:09.910281+0000 mgr.y (mgr.14140) 419 : cluster [DBG] pgmap v407: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 91 KiB/s rd, 170 B/s wr, 164 op/s 2024-02-10T09:29:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:13 smithi204 bash[35809]: cluster 2024-02-10T09:29:11.910862+0000 mgr.y (mgr.14140) 420 : cluster [DBG] pgmap v408: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:13 smithi153 bash[35780]: cluster 2024-02-10T09:29:11.910862+0000 mgr.y (mgr.14140) 420 : cluster [DBG] pgmap v408: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:13 smithi153 bash[39117]: cluster 2024-02-10T09:29:11.910862+0000 mgr.y (mgr.14140) 420 : cluster [DBG] pgmap v408: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:16.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:15 smithi204 bash[35809]: cluster 2024-02-10T09:29:13.911530+0000 mgr.y (mgr.14140) 421 : cluster [DBG] pgmap v409: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:15 smithi153 bash[35780]: cluster 2024-02-10T09:29:13.911530+0000 mgr.y (mgr.14140) 421 : cluster [DBG] pgmap v409: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:15 smithi153 bash[39117]: cluster 2024-02-10T09:29:13.911530+0000 mgr.y (mgr.14140) 421 : cluster [DBG] pgmap v409: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:17 smithi204 bash[35809]: cluster 2024-02-10T09:29:15.912237+0000 mgr.y (mgr.14140) 422 : cluster [DBG] pgmap v410: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:17 smithi153 bash[35780]: cluster 2024-02-10T09:29:15.912237+0000 mgr.y (mgr.14140) 422 : cluster [DBG] pgmap v410: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:17 smithi153 bash[39117]: cluster 2024-02-10T09:29:15.912237+0000 mgr.y (mgr.14140) 422 : cluster [DBG] pgmap v410: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 166 op/s 2024-02-10T09:29:20.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:19 smithi204 bash[35809]: cluster 2024-02-10T09:29:17.912844+0000 mgr.y (mgr.14140) 423 : cluster [DBG] pgmap v411: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 58 KiB/s rd, 170 B/s wr, 100 op/s 2024-02-10T09:29:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:19 smithi153 bash[35780]: cluster 2024-02-10T09:29:17.912844+0000 mgr.y (mgr.14140) 423 : cluster [DBG] pgmap v411: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 58 KiB/s rd, 170 B/s wr, 100 op/s 2024-02-10T09:29:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:19 smithi153 bash[39117]: cluster 2024-02-10T09:29:17.912844+0000 mgr.y (mgr.14140) 423 : cluster [DBG] pgmap v411: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 58 KiB/s rd, 170 B/s wr, 100 op/s 2024-02-10T09:29:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:21 smithi204 bash[35809]: cluster 2024-02-10T09:29:19.913386+0000 mgr.y (mgr.14140) 424 : cluster [DBG] pgmap v412: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 75 op/s 2024-02-10T09:29:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:21 smithi153 bash[35780]: cluster 2024-02-10T09:29:19.913386+0000 mgr.y (mgr.14140) 424 : cluster [DBG] pgmap v412: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 75 op/s 2024-02-10T09:29:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:21 smithi153 bash[39117]: cluster 2024-02-10T09:29:19.913386+0000 mgr.y (mgr.14140) 424 : cluster [DBG] pgmap v412: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 75 op/s 2024-02-10T09:29:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:23 smithi204 bash[35809]: cluster 2024-02-10T09:29:21.913963+0000 mgr.y (mgr.14140) 425 : cluster [DBG] pgmap v413: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T09:29:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:23 smithi153 bash[35780]: cluster 2024-02-10T09:29:21.913963+0000 mgr.y (mgr.14140) 425 : cluster [DBG] pgmap v413: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T09:29:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:23 smithi153 bash[39117]: cluster 2024-02-10T09:29:21.913963+0000 mgr.y (mgr.14140) 425 : cluster [DBG] pgmap v413: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T09:29:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:25 smithi204 bash[35809]: cluster 2024-02-10T09:29:23.914598+0000 mgr.y (mgr.14140) 426 : cluster [DBG] pgmap v414: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:25 smithi153 bash[35780]: cluster 2024-02-10T09:29:23.914598+0000 mgr.y (mgr.14140) 426 : cluster [DBG] pgmap v414: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:25 smithi153 bash[39117]: cluster 2024-02-10T09:29:23.914598+0000 mgr.y (mgr.14140) 426 : cluster [DBG] pgmap v414: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:27 smithi204 bash[35809]: cluster 2024-02-10T09:29:25.915165+0000 mgr.y (mgr.14140) 427 : cluster [DBG] pgmap v415: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:27 smithi153 bash[35780]: cluster 2024-02-10T09:29:25.915165+0000 mgr.y (mgr.14140) 427 : cluster [DBG] pgmap v415: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:27 smithi153 bash[39117]: cluster 2024-02-10T09:29:25.915165+0000 mgr.y (mgr.14140) 427 : cluster [DBG] pgmap v415: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:30.006 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:29 smithi153 bash[35780]: cluster 2024-02-10T09:29:27.915769+0000 mgr.y (mgr.14140) 428 : cluster [DBG] pgmap v416: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:30.006 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:29 smithi153 bash[39117]: cluster 2024-02-10T09:29:27.915769+0000 mgr.y (mgr.14140) 428 : cluster [DBG] pgmap v416: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:30.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:29 smithi204 bash[35809]: cluster 2024-02-10T09:29:27.915769+0000 mgr.y (mgr.14140) 428 : cluster [DBG] pgmap v416: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:30.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:29:30 smithi153 bash[55964]: time="2024-02-10T09:29:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:29:31.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:29:30 smithi204 bash[45101]: time="2024-02-10T09:29:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:29:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:31 smithi204 bash[35809]: cluster 2024-02-10T09:29:29.916322+0000 mgr.y (mgr.14140) 429 : cluster [DBG] pgmap v417: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:31 smithi153 bash[35780]: cluster 2024-02-10T09:29:29.916322+0000 mgr.y (mgr.14140) 429 : cluster [DBG] pgmap v417: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:31 smithi153 bash[39117]: cluster 2024-02-10T09:29:29.916322+0000 mgr.y (mgr.14140) 429 : cluster [DBG] pgmap v417: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:33 smithi204 bash[35809]: cluster 2024-02-10T09:29:31.917053+0000 mgr.y (mgr.14140) 430 : cluster [DBG] pgmap v418: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:34.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:33 smithi153 bash[35780]: cluster 2024-02-10T09:29:31.917053+0000 mgr.y (mgr.14140) 430 : cluster [DBG] pgmap v418: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:33 smithi153 bash[39117]: cluster 2024-02-10T09:29:31.917053+0000 mgr.y (mgr.14140) 430 : cluster [DBG] pgmap v418: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:36.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:35 smithi204 bash[35809]: cluster 2024-02-10T09:29:33.917626+0000 mgr.y (mgr.14140) 431 : cluster [DBG] pgmap v419: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:35 smithi153 bash[35780]: cluster 2024-02-10T09:29:33.917626+0000 mgr.y (mgr.14140) 431 : cluster [DBG] pgmap v419: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:35 smithi153 bash[39117]: cluster 2024-02-10T09:29:33.917626+0000 mgr.y (mgr.14140) 431 : cluster [DBG] pgmap v419: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:37 smithi204 bash[35809]: cluster 2024-02-10T09:29:35.918219+0000 mgr.y (mgr.14140) 432 : cluster [DBG] pgmap v420: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:37 smithi153 bash[35780]: cluster 2024-02-10T09:29:35.918219+0000 mgr.y (mgr.14140) 432 : cluster [DBG] pgmap v420: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:37 smithi153 bash[39117]: cluster 2024-02-10T09:29:35.918219+0000 mgr.y (mgr.14140) 432 : cluster [DBG] pgmap v420: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:39 smithi204 bash[35809]: cluster 2024-02-10T09:29:37.918671+0000 mgr.y (mgr.14140) 433 : cluster [DBG] pgmap v421: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:39 smithi153 bash[35780]: cluster 2024-02-10T09:29:37.918671+0000 mgr.y (mgr.14140) 433 : cluster [DBG] pgmap v421: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:39 smithi153 bash[39117]: cluster 2024-02-10T09:29:37.918671+0000 mgr.y (mgr.14140) 433 : cluster [DBG] pgmap v421: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:41 smithi204 bash[35809]: cluster 2024-02-10T09:29:39.919208+0000 mgr.y (mgr.14140) 434 : cluster [DBG] pgmap v422: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:41 smithi153 bash[35780]: cluster 2024-02-10T09:29:39.919208+0000 mgr.y (mgr.14140) 434 : cluster [DBG] pgmap v422: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:41 smithi153 bash[39117]: cluster 2024-02-10T09:29:39.919208+0000 mgr.y (mgr.14140) 434 : cluster [DBG] pgmap v422: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:43 smithi204 bash[35809]: cluster 2024-02-10T09:29:41.919828+0000 mgr.y (mgr.14140) 435 : cluster [DBG] pgmap v423: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:43 smithi153 bash[35780]: cluster 2024-02-10T09:29:41.919828+0000 mgr.y (mgr.14140) 435 : cluster [DBG] pgmap v423: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:43 smithi153 bash[39117]: cluster 2024-02-10T09:29:41.919828+0000 mgr.y (mgr.14140) 435 : cluster [DBG] pgmap v423: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:45 smithi204 bash[35809]: cluster 2024-02-10T09:29:43.920583+0000 mgr.y (mgr.14140) 436 : cluster [DBG] pgmap v424: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:45 smithi153 bash[35780]: cluster 2024-02-10T09:29:43.920583+0000 mgr.y (mgr.14140) 436 : cluster [DBG] pgmap v424: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:45 smithi153 bash[39117]: cluster 2024-02-10T09:29:43.920583+0000 mgr.y (mgr.14140) 436 : cluster [DBG] pgmap v424: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:48.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:47 smithi204 bash[35809]: cluster 2024-02-10T09:29:45.921226+0000 mgr.y (mgr.14140) 437 : cluster [DBG] pgmap v425: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:47 smithi153 bash[35780]: cluster 2024-02-10T09:29:45.921226+0000 mgr.y (mgr.14140) 437 : cluster [DBG] pgmap v425: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:47 smithi153 bash[39117]: cluster 2024-02-10T09:29:45.921226+0000 mgr.y (mgr.14140) 437 : cluster [DBG] pgmap v425: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:49 smithi204 bash[35809]: cluster 2024-02-10T09:29:47.921785+0000 mgr.y (mgr.14140) 438 : cluster [DBG] pgmap v426: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:49 smithi153 bash[35780]: cluster 2024-02-10T09:29:47.921785+0000 mgr.y (mgr.14140) 438 : cluster [DBG] pgmap v426: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:49 smithi153 bash[39117]: cluster 2024-02-10T09:29:47.921785+0000 mgr.y (mgr.14140) 438 : cluster [DBG] pgmap v426: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:51 smithi204 bash[35809]: cluster 2024-02-10T09:29:49.922355+0000 mgr.y (mgr.14140) 439 : cluster [DBG] pgmap v427: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:51 smithi153 bash[35780]: cluster 2024-02-10T09:29:49.922355+0000 mgr.y (mgr.14140) 439 : cluster [DBG] pgmap v427: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:51 smithi153 bash[39117]: cluster 2024-02-10T09:29:49.922355+0000 mgr.y (mgr.14140) 439 : cluster [DBG] pgmap v427: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:53 smithi204 bash[35809]: cluster 2024-02-10T09:29:51.922998+0000 mgr.y (mgr.14140) 440 : cluster [DBG] pgmap v428: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:53 smithi153 bash[39117]: cluster 2024-02-10T09:29:51.922998+0000 mgr.y (mgr.14140) 440 : cluster [DBG] pgmap v428: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:53 smithi153 bash[35780]: cluster 2024-02-10T09:29:51.922998+0000 mgr.y (mgr.14140) 440 : cluster [DBG] pgmap v428: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:56.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:55 smithi204 bash[35809]: cluster 2024-02-10T09:29:53.923738+0000 mgr.y (mgr.14140) 441 : cluster [DBG] pgmap v429: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:55 smithi153 bash[35780]: cluster 2024-02-10T09:29:53.923738+0000 mgr.y (mgr.14140) 441 : cluster [DBG] pgmap v429: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:56.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:55 smithi153 bash[39117]: cluster 2024-02-10T09:29:53.923738+0000 mgr.y (mgr.14140) 441 : cluster [DBG] pgmap v429: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:58.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:57 smithi204 bash[35809]: cluster 2024-02-10T09:29:55.924458+0000 mgr.y (mgr.14140) 442 : cluster [DBG] pgmap v430: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:57 smithi153 bash[35780]: cluster 2024-02-10T09:29:55.924458+0000 mgr.y (mgr.14140) 442 : cluster [DBG] pgmap v430: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:29:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:57 smithi153 bash[39117]: cluster 2024-02-10T09:29:55.924458+0000 mgr.y (mgr.14140) 442 : cluster [DBG] pgmap v430: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:29:59 smithi204 bash[35809]: cluster 2024-02-10T09:29:57.925110+0000 mgr.y (mgr.14140) 443 : cluster [DBG] pgmap v431: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:29:59 smithi153 bash[35780]: cluster 2024-02-10T09:29:57.925110+0000 mgr.y (mgr.14140) 443 : cluster [DBG] pgmap v431: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:29:59 smithi153 bash[39117]: cluster 2024-02-10T09:29:57.925110+0000 mgr.y (mgr.14140) 443 : cluster [DBG] pgmap v431: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:01 smithi204 bash[35809]: cluster 2024-02-10T09:29:59.925771+0000 mgr.y (mgr.14140) 444 : cluster [DBG] pgmap v432: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:01 smithi153 bash[39117]: cluster 2024-02-10T09:29:59.925771+0000 mgr.y (mgr.14140) 444 : cluster [DBG] pgmap v432: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:01 smithi153 bash[35780]: cluster 2024-02-10T09:29:59.925771+0000 mgr.y (mgr.14140) 444 : cluster [DBG] pgmap v432: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:03 smithi153 bash[35780]: cluster 2024-02-10T09:30:01.926272+0000 mgr.y (mgr.14140) 445 : cluster [DBG] pgmap v433: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:03 smithi153 bash[39117]: cluster 2024-02-10T09:30:01.926272+0000 mgr.y (mgr.14140) 445 : cluster [DBG] pgmap v433: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:03 smithi204 bash[35809]: cluster 2024-02-10T09:30:01.926272+0000 mgr.y (mgr.14140) 445 : cluster [DBG] pgmap v433: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:05.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:30:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:05 smithi153 bash[35780]: cluster 2024-02-10T09:30:03.926932+0000 mgr.y (mgr.14140) 446 : cluster [DBG] pgmap v434: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:05 smithi153 bash[39117]: cluster 2024-02-10T09:30:03.926932+0000 mgr.y (mgr.14140) 446 : cluster [DBG] pgmap v434: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:05 smithi204 bash[35809]: cluster 2024-02-10T09:30:03.926932+0000 mgr.y (mgr.14140) 446 : cluster [DBG] pgmap v434: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (66s) 57s ago 66s 6.6.2 docker.io/ceph/ceph-grafana:latest 87a51ecf0b1c a148afe5b831 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (3m) 57s ago 11m 16.2.14-521-g0e714d9a quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 1fbc92c859dc 46c6073a79be 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 running (13m) 61s ago 13m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 03548f93e5ac 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (13m) 61s ago 13m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (11m) 57s ago 11m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:30:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (12m) 61s ago 12m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running 61s ago 3m prom/node-exporter 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running 57s ago 3m prom/node-exporter 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (10m) 61s ago 11m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (10m) 61s ago 10m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (10m) 61s ago 10m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (10m) 61s ago 10m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (9m) 57s ago 9m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (9m) 57s ago 9m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (9m) 57s ago 9m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (8m) 57s ago 9m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (64s) 61s ago 64s 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:30:06.996 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (62s) 57s ago 62s 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:30:07.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:30:07 smithi153 bash[55964]: time="2024-02-10T09:30:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:30:07.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:30:07.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:07 smithi153 bash[35780]: cluster 2024-02-10T09:30:05.927579+0000 mgr.y (mgr.14140) 447 : cluster [DBG] pgmap v435: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:07 smithi153 bash[39117]: cluster 2024-02-10T09:30:05.927579+0000 mgr.y (mgr.14140) 447 : cluster [DBG] pgmap v435: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:08.096 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:07 smithi204 bash[35809]: cluster 2024-02-10T09:30:05.927579+0000 mgr.y (mgr.14140) 447 : cluster [DBG] pgmap v435: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:08.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:30:08 smithi204 bash[45101]: time="2024-02-10T09:30:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:30:08.874 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:08 smithi153 bash[35780]: audit 2024-02-10T09:30:06.991165+0000 mgr.y (mgr.14140) 448 : audit [DBG] from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:30:08.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:08 smithi153 bash[39117]: audit 2024-02-10T09:30:06.991165+0000 mgr.y (mgr.14140) 448 : audit [DBG] from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:30:09.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:08 smithi204 bash[35809]: audit 2024-02-10T09:30:06.991165+0000 mgr.y (mgr.14140) 448 : audit [DBG] from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:30:09.554 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 1, 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 1 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:30:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:30:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:30:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 14, 2024-02-10T09:30:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 1 2024-02-10T09:30:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:30:09.556 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:30:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:09 smithi153 bash[35780]: cluster 2024-02-10T09:30:07.927974+0000 mgr.y (mgr.14140) 449 : cluster [DBG] pgmap v436: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:09 smithi153 bash[35780]: audit 2024-02-10T09:30:09.554032+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.153:0/3483932237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:30:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:09 smithi153 bash[39117]: cluster 2024-02-10T09:30:07.927974+0000 mgr.y (mgr.14140) 449 : cluster [DBG] pgmap v436: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:09 smithi153 bash[39117]: audit 2024-02-10T09:30:09.554032+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.153:0/3483932237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:30:10.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph -s' 2024-02-10T09:30:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:09 smithi204 bash[35809]: cluster 2024-02-10T09:30:07.927974+0000 mgr.y (mgr.14140) 449 : cluster [DBG] pgmap v436: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:09 smithi204 bash[35809]: audit 2024-02-10T09:30:09.554032+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.153:0/3483932237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: cluster: 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: id: 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: health: HEALTH_OK 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: services: 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: mgr: y(active, since 12m), standbys: x 2024-02-10T09:30:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: rgw: 2 daemons active (r.z.smithi153.zyveqt, r.z.smithi204.kmhqqg) 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: task status: 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: data: 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: pools: 5 pools, 129 pgs 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: objects: 202 objects, 7.0 KiB 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: usage: 8.0 GiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: pgs: 129 active+clean 2024-02-10T09:30:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:30:12.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:11 smithi153 bash[35780]: cluster 2024-02-10T09:30:09.928433+0000 mgr.y (mgr.14140) 450 : cluster [DBG] pgmap v437: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:11 smithi153 bash[39117]: cluster 2024-02-10T09:30:09.928433+0000 mgr.y (mgr.14140) 450 : cluster [DBG] pgmap v437: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:11 smithi204 bash[35809]: cluster 2024-02-10T09:30:09.928433+0000 mgr.y (mgr.14140) 450 : cluster [DBG] pgmap v437: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:12.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-02-10T09:30:13.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:12 smithi153 bash[35780]: audit 2024-02-10T09:30:12.180179+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.153:0/1224838862' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:30:13.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:12 smithi153 bash[39117]: audit 2024-02-10T09:30:12.180179+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.153:0/1224838862' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:30:13.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:12 smithi204 bash[35809]: audit 2024-02-10T09:30:12.180179+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.153:0/1224838862' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:30:14.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:13 smithi153 bash[39117]: cluster 2024-02-10T09:30:11.928906+0000 mgr.y (mgr.14140) 451 : cluster [DBG] pgmap v438: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:14.147 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:13 smithi153 bash[35780]: cluster 2024-02-10T09:30:11.928906+0000 mgr.y (mgr.14140) 451 : cluster [DBG] pgmap v438: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:13 smithi204 bash[35809]: cluster 2024-02-10T09:30:11.928906+0000 mgr.y (mgr.14140) 451 : cluster [DBG] pgmap v438: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:14.819 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:30:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:14 smithi153 bash[35780]: audit 2024-02-10T09:30:14.808445+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.153:0/1119317367' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:30:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:14 smithi153 bash[39117]: audit 2024-02-10T09:30:14.808445+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.153:0/1119317367' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:30:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:14 smithi204 bash[35809]: audit 2024-02-10T09:30:14.808445+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.153:0/1119317367' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:30:15.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph mgr fail' 2024-02-10T09:30:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:15 smithi153 bash[35780]: cluster 2024-02-10T09:30:13.929430+0000 mgr.y (mgr.14140) 452 : cluster [DBG] pgmap v439: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:15 smithi153 bash[39117]: cluster 2024-02-10T09:30:13.929430+0000 mgr.y (mgr.14140) 452 : cluster [DBG] pgmap v439: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:15 smithi204 bash[35809]: cluster 2024-02-10T09:30:13.929430+0000 mgr.y (mgr.14140) 452 : cluster [DBG] pgmap v439: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:18.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[36014]: ignoring --setuser ceph since I am not root 2024-02-10T09:30:18.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[36014]: ignoring --setgroup ceph since I am not root 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[35780]: cluster 2024-02-10T09:30:15.929971+0000 mgr.y (mgr.14140) 453 : cluster [DBG] pgmap v440: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[35780]: audit 2024-02-10T09:30:17.571450+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.153:0/2088213301' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[35780]: audit 2024-02-10T09:30:17.571741+0000 mon.a (mon.0) 623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[35780]: cluster 2024-02-10T09:30:17.576637+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[39117]: cluster 2024-02-10T09:30:15.929971+0000 mgr.y (mgr.14140) 453 : cluster [DBG] pgmap v440: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[39117]: audit 2024-02-10T09:30:17.571450+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.153:0/2088213301' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[39117]: audit 2024-02-10T09:30:17.571741+0000 mon.a (mon.0) 623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:30:18.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:17 smithi153 bash[39117]: cluster 2024-02-10T09:30:17.576637+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-10T09:30:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:17 smithi204 bash[35809]: cluster 2024-02-10T09:30:15.929971+0000 mgr.y (mgr.14140) 453 : cluster [DBG] pgmap v440: 129 pgs: 129 active+clean; 7.0 KiB data, 5.5 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:17 smithi204 bash[35809]: audit 2024-02-10T09:30:17.571450+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.153:0/2088213301' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:30:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:17 smithi204 bash[35809]: audit 2024-02-10T09:30:17.571741+0000 mon.a (mon.0) 623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:30:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:17 smithi204 bash[35809]: cluster 2024-02-10T09:30:17.576637+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-10T09:30:18.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 180' 2024-02-10T09:30:18.892 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:18 smithi153 bash[35780]: audit 2024-02-10T09:30:17.886882+0000 mon.a (mon.0) 625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:30:18.892 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:18 smithi153 bash[35780]: cluster 2024-02-10T09:30:17.886937+0000 mon.a (mon.0) 626 : cluster [DBG] mgrmap e16: x(active, starting, since 0.31477s) 2024-02-10T09:30:19.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:18 smithi153 bash[39117]: audit 2024-02-10T09:30:17.886882+0000 mon.a (mon.0) 625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:30:19.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:18 smithi153 bash[39117]: cluster 2024-02-10T09:30:17.886937+0000 mon.a (mon.0) 626 : cluster [DBG] mgrmap e16: x(active, starting, since 0.31477s) 2024-02-10T09:30:19.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:18 smithi204 bash[35809]: audit 2024-02-10T09:30:17.886882+0000 mon.a (mon.0) 625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:30:19.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:18 smithi204 bash[35809]: cluster 2024-02-10T09:30:17.886937+0000 mon.a (mon.0) 626 : cluster [DBG] mgrmap e16: x(active, starting, since 0.31477s) 2024-02-10T09:30:21.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:20 smithi153 bash[39117]: cluster 2024-02-10T09:30:19.896897+0000 mon.a (mon.0) 627 : cluster [DBG] mgrmap e17: x(active, starting, since 2s) 2024-02-10T09:30:21.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:20 smithi153 bash[39117]: audit 2024-02-10T09:30:19.898047+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:30:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:20 smithi153 bash[35780]: cluster 2024-02-10T09:30:19.896897+0000 mon.a (mon.0) 627 : cluster [DBG] mgrmap e17: x(active, starting, since 2s) 2024-02-10T09:30:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:20 smithi153 bash[35780]: audit 2024-02-10T09:30:19.898047+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:30:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:20 smithi204 bash[35809]: cluster 2024-02-10T09:30:19.896897+0000 mon.a (mon.0) 627 : cluster [DBG] mgrmap e17: x(active, starting, since 2s) 2024-02-10T09:30:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:20 smithi204 bash[35809]: audit 2024-02-10T09:30:19.898047+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:30:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:21 smithi153 bash[35780]: cluster 2024-02-10T09:30:21.589260+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:30:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:21 smithi153 bash[35780]: audit 2024-02-10T09:30:21.589645+0000 mon.c (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.153:0/1124429987' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:30:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:21 smithi153 bash[35780]: audit 2024-02-10T09:30:21.590665+0000 mon.c (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.153:0/1124429987' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:30:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:21 smithi153 bash[39117]: cluster 2024-02-10T09:30:21.589260+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:30:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:21 smithi153 bash[39117]: audit 2024-02-10T09:30:21.589645+0000 mon.c (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.153:0/1124429987' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:30:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:21 smithi153 bash[39117]: audit 2024-02-10T09:30:21.590665+0000 mon.c (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.153:0/1124429987' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:30:22.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:21 smithi204 bash[35809]: cluster 2024-02-10T09:30:21.589260+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:30:22.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:21 smithi204 bash[35809]: audit 2024-02-10T09:30:21.589645+0000 mon.c (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.153:0/1124429987' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:30:22.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:21 smithi204 bash[35809]: audit 2024-02-10T09:30:21.590665+0000 mon.c (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.153:0/1124429987' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:30:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:22 smithi153 bash[35780]: cluster 2024-02-10T09:30:21.911051+0000 mon.a (mon.0) 629 : cluster [DBG] mgrmap e18: x(active, starting, since 4s), standbys: y 2024-02-10T09:30:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:22 smithi153 bash[35780]: audit 2024-02-10T09:30:21.913303+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:30:23.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:22 smithi153 bash[39117]: cluster 2024-02-10T09:30:21.911051+0000 mon.a (mon.0) 629 : cluster [DBG] mgrmap e18: x(active, starting, since 4s), standbys: y 2024-02-10T09:30:23.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:22 smithi153 bash[39117]: audit 2024-02-10T09:30:21.913303+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:30:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:22 smithi204 bash[35809]: cluster 2024-02-10T09:30:21.911051+0000 mon.a (mon.0) 629 : cluster [DBG] mgrmap e18: x(active, starting, since 4s), standbys: y 2024-02-10T09:30:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:22 smithi204 bash[35809]: audit 2024-02-10T09:30:21.913303+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.888457+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.888705+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.888942+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.890286+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.890590+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.890864+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.891156+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.891420+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.891685+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:30:24.169 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.891954+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.892252+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.893449+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.893710+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.894374+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.895157+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.895975+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.896272+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.896592+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.888457+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:30:24.170 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.888705+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.888942+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.890286+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.890590+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.890864+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.891156+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.891420+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.891685+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.891954+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.892252+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.893449+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.893710+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.894374+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.895157+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.895975+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.896272+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:30:24.171 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.896592+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.896882+0000 mon.b (mon.2) 45 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.897181+0000 mon.b (mon.2) 46 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.897471+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.897752+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.898022+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.898299+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.898556+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.898810+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.899087+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.899382+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:30:24.172 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.899627+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.899872+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.900116+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.900384+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.900633+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.900878+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[35780]: audit 2024-02-10T09:30:23.901120+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.896882+0000 mon.b (mon.2) 45 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.897181+0000 mon.b (mon.2) 46 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.897471+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.897752+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.898022+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.898299+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.898556+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:30:24.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.898810+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.899087+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.899382+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.899627+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.899872+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.900116+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.900384+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.900633+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.900878+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:30:24.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:23 smithi153 bash[39117]: audit 2024-02-10T09:30:23.901120+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:30:24.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[44646]: Warning: Permanently added '172.21.15.153' (ECDSA) to the list of known hosts. 2024-02-10T09:30:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.888457+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.888705+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.888942+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.890286+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.890590+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.890864+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.891156+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.891420+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.891685+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.891954+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.892252+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.893449+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.893710+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:30:24.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.894374+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.895157+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.895975+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.896272+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.896592+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.896882+0000 mon.b (mon.2) 45 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.897181+0000 mon.b (mon.2) 46 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.897471+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.897752+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.898022+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.898299+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.898556+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.898810+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.899087+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.899382+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.899627+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.899872+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.900116+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.900384+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.900633+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.900878+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:23 smithi204 bash[35809]: audit 2024-02-10T09:30:23.901120+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: cluster 2024-02-10T09:30:23.928440+0000 mon.a (mon.0) 630 : cluster [INF] Manager daemon x is now available 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: cephadm 2024-02-10T09:30:23.939883+0000 mgr.x (mgr.34547) 1 : cephadm [INF] inventory: adjusted host smithi153 addr 'smithi153' -> '172.21.15.153' 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:23.940324+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:23.940838+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:23.943726+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: cephadm 2024-02-10T09:30:23.948388+0000 mgr.x (mgr.34547) 2 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:23.948687+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]: dispatch 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:23.949339+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]: dispatch 2024-02-10T09:30:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:23.952834+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]': finished 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.040616+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]: dispatch 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.041013+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]: dispatch 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: cluster 2024-02-10T09:30:23.928440+0000 mon.a (mon.0) 630 : cluster [INF] Manager daemon x is now available 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: cephadm 2024-02-10T09:30:23.939883+0000 mgr.x (mgr.34547) 1 : cephadm [INF] inventory: adjusted host smithi153 addr 'smithi153' -> '172.21.15.153' 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:23.940324+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:23.940838+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:23.943726+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: cephadm 2024-02-10T09:30:23.948388+0000 mgr.x (mgr.34547) 2 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:23.948687+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]: dispatch 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:23.949339+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]: dispatch 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:23.952834+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]': finished 2024-02-10T09:30:25.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.040616+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.041013+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.043666+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]': finished 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.045057+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.046410+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.075738+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.076224+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.080821+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[39117]: audit 2024-02-10T09:30:24.081272+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.043666+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]': finished 2024-02-10T09:30:25.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.045057+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.046410+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.075738+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:25.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.076224+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:25.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.080821+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:25.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:24 smithi153 bash[35780]: audit 2024-02-10T09:30:24.081272+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: cluster 2024-02-10T09:30:23.928440+0000 mon.a (mon.0) 630 : cluster [INF] Manager daemon x is now available 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: cephadm 2024-02-10T09:30:23.939883+0000 mgr.x (mgr.34547) 1 : cephadm [INF] inventory: adjusted host smithi153 addr 'smithi153' -> '172.21.15.153' 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:23.940324+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:23.940838+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:23.943726+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"smithi204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: cephadm 2024-02-10T09:30:23.948388+0000 mgr.x (mgr.34547) 2 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:23.948687+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]: dispatch 2024-02-10T09:30:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:23.949339+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:23.952834+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"5","who":"mgr","name":"mgr/cephadm/migration_current"}]': finished 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.040616+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.041013+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.043666+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/config_checks","val":"{\"kernel_security\": \"enabled\", \"os_subscription\": \"enabled\", \"public_network\": \"enabled\", \"osd_mtu_size\": \"enabled\", \"osd_linkspeed\": \"enabled\", \"network_missing\": \"enabled\", \"ceph_release\": \"enabled\", \"kernel_version\": \"enabled\"}"}]': finished 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.045057+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.046410+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.075738+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.076224+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.080821+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:25.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:24 smithi204 bash[35809]: audit 2024-02-10T09:30:24.081272+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:25 smithi153 bash[39117]: cluster 2024-02-10T09:30:24.925953+0000 mon.a (mon.0) 639 : cluster [DBG] mgrmap e19: x(active, since 7s), standbys: y 2024-02-10T09:30:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:25 smithi153 bash[39117]: cluster 2024-02-10T09:30:24.936684+0000 mgr.x (mgr.34547) 3 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:25 smithi153 bash[35780]: cluster 2024-02-10T09:30:24.925953+0000 mon.a (mon.0) 639 : cluster [DBG] mgrmap e19: x(active, since 7s), standbys: y 2024-02-10T09:30:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:25 smithi153 bash[35780]: cluster 2024-02-10T09:30:24.936684+0000 mgr.x (mgr.34547) 3 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:25 smithi204 bash[35809]: cluster 2024-02-10T09:30:24.925953+0000 mon.a (mon.0) 639 : cluster [DBG] mgrmap e19: x(active, since 7s), standbys: y 2024-02-10T09:30:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:25 smithi204 bash[35809]: cluster 2024-02-10T09:30:24.936684+0000 mgr.x (mgr.34547) 3 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:26 smithi153 bash[39117]: cluster 2024-02-10T09:30:25.893608+0000 mgr.x (mgr.34547) 4 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:26 smithi153 bash[39117]: cluster 2024-02-10T09:30:25.931560+0000 mon.a (mon.0) 640 : cluster [DBG] mgrmap e20: x(active, since 8s), standbys: y 2024-02-10T09:30:27.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:26 smithi153 bash[35780]: cluster 2024-02-10T09:30:25.893608+0000 mgr.x (mgr.34547) 4 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:27.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:26 smithi153 bash[35780]: cluster 2024-02-10T09:30:25.931560+0000 mon.a (mon.0) 640 : cluster [DBG] mgrmap e20: x(active, since 8s), standbys: y 2024-02-10T09:30:27.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:26 smithi204 bash[35809]: cluster 2024-02-10T09:30:25.893608+0000 mgr.x (mgr.34547) 4 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:27.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:26 smithi204 bash[35809]: cluster 2024-02-10T09:30:25.931560+0000 mon.a (mon.0) 640 : cluster [DBG] mgrmap e20: x(active, since 8s), standbys: y 2024-02-10T09:30:29.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:28 smithi153 bash[35780]: cluster 2024-02-10T09:30:27.894219+0000 mgr.x (mgr.34547) 5 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:28 smithi153 bash[39117]: cluster 2024-02-10T09:30:27.894219+0000 mgr.x (mgr.34547) 5 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:28 smithi204 bash[35809]: cluster 2024-02-10T09:30:27.894219+0000 mgr.x (mgr.34547) 5 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:30 smithi204 bash[35809]: cluster 2024-02-10T09:30:29.894730+0000 mgr.x (mgr.34547) 6 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:31.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:30 smithi153 bash[35780]: cluster 2024-02-10T09:30:29.894730+0000 mgr.x (mgr.34547) 6 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:31.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:30 smithi153 bash[39117]: cluster 2024-02-10T09:30:29.894730+0000 mgr.x (mgr.34547) 6 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:33.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:32 smithi204 bash[35809]: cluster 2024-02-10T09:30:31.895478+0000 mgr.x (mgr.34547) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:33.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:32 smithi153 bash[35780]: cluster 2024-02-10T09:30:31.895478+0000 mgr.x (mgr.34547) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:33.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:32 smithi153 bash[39117]: cluster 2024-02-10T09:30:31.895478+0000 mgr.x (mgr.34547) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:34 smithi204 bash[35809]: cluster 2024-02-10T09:30:33.896067+0000 mgr.x (mgr.34547) 8 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:34 smithi153 bash[35780]: cluster 2024-02-10T09:30:33.896067+0000 mgr.x (mgr.34547) 8 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:35.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:34 smithi153 bash[39117]: cluster 2024-02-10T09:30:33.896067+0000 mgr.x (mgr.34547) 8 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:36 smithi204 bash[35809]: cluster 2024-02-10T09:30:35.896713+0000 mgr.x (mgr.34547) 9 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:37.438 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:36 smithi153 bash[39117]: cluster 2024-02-10T09:30:35.896713+0000 mgr.x (mgr.34547) 9 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:37.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:36 smithi153 bash[35780]: cluster 2024-02-10T09:30:35.896713+0000 mgr.x (mgr.34547) 9 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:39.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:38 smithi204 bash[35809]: cluster 2024-02-10T09:30:37.897087+0000 mgr.x (mgr.34547) 10 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:39.438 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:38 smithi153 bash[39117]: cluster 2024-02-10T09:30:37.897087+0000 mgr.x (mgr.34547) 10 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:38 smithi153 bash[35780]: cluster 2024-02-10T09:30:37.897087+0000 mgr.x (mgr.34547) 10 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:41.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:40 smithi204 bash[35809]: cluster 2024-02-10T09:30:39.897638+0000 mgr.x (mgr.34547) 11 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:40 smithi153 bash[35780]: cluster 2024-02-10T09:30:39.897638+0000 mgr.x (mgr.34547) 11 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:41.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:40 smithi153 bash[39117]: cluster 2024-02-10T09:30:39.897638+0000 mgr.x (mgr.34547) 11 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:43.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:42 smithi204 bash[35809]: cluster 2024-02-10T09:30:41.898325+0000 mgr.x (mgr.34547) 12 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:43.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:42 smithi153 bash[35780]: cluster 2024-02-10T09:30:41.898325+0000 mgr.x (mgr.34547) 12 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:43.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:42 smithi153 bash[39117]: cluster 2024-02-10T09:30:41.898325+0000 mgr.x (mgr.34547) 12 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:44 smithi204 bash[35809]: cluster 2024-02-10T09:30:43.898740+0000 mgr.x (mgr.34547) 13 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:44 smithi153 bash[39117]: cluster 2024-02-10T09:30:43.898740+0000 mgr.x (mgr.34547) 13 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:44 smithi153 bash[35780]: cluster 2024-02-10T09:30:43.898740+0000 mgr.x (mgr.34547) 13 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:45.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:30:44 smithi153 bash[55964]: time="2024-02-10T09:30:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:30:46.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:30:45 smithi204 bash[45101]: time="2024-02-10T09:30:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:30:47.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:46 smithi204 bash[35809]: cluster 2024-02-10T09:30:45.899649+0000 mgr.x (mgr.34547) 14 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:47.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:46 smithi153 bash[35780]: cluster 2024-02-10T09:30:45.899649+0000 mgr.x (mgr.34547) 14 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:46 smithi153 bash[39117]: cluster 2024-02-10T09:30:45.899649+0000 mgr.x (mgr.34547) 14 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:49.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:48 smithi153 bash[35780]: cluster 2024-02-10T09:30:47.900215+0000 mgr.x (mgr.34547) 15 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:48 smithi153 bash[39117]: cluster 2024-02-10T09:30:47.900215+0000 mgr.x (mgr.34547) 15 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:49.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:48 smithi204 bash[35809]: cluster 2024-02-10T09:30:47.900215+0000 mgr.x (mgr.34547) 15 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:51.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:50 smithi204 bash[35809]: cluster 2024-02-10T09:30:49.900631+0000 mgr.x (mgr.34547) 16 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:51.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:50 smithi153 bash[35780]: cluster 2024-02-10T09:30:49.900631+0000 mgr.x (mgr.34547) 16 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:51.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:50 smithi153 bash[39117]: cluster 2024-02-10T09:30:49.900631+0000 mgr.x (mgr.34547) 16 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:53.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:52 smithi204 bash[35809]: cluster 2024-02-10T09:30:51.901353+0000 mgr.x (mgr.34547) 17 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:53.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:52 smithi153 bash[35780]: cluster 2024-02-10T09:30:51.901353+0000 mgr.x (mgr.34547) 17 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:52 smithi153 bash[39117]: cluster 2024-02-10T09:30:51.901353+0000 mgr.x (mgr.34547) 17 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:54 smithi204 bash[35809]: cluster 2024-02-10T09:30:53.902001+0000 mgr.x (mgr.34547) 18 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:54 smithi153 bash[35780]: cluster 2024-02-10T09:30:53.902001+0000 mgr.x (mgr.34547) 18 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:55.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:54 smithi153 bash[39117]: cluster 2024-02-10T09:30:53.902001+0000 mgr.x (mgr.34547) 18 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:57.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:56 smithi153 bash[35780]: cluster 2024-02-10T09:30:55.902722+0000 mgr.x (mgr.34547) 19 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:57.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:56 smithi153 bash[39117]: cluster 2024-02-10T09:30:55.902722+0000 mgr.x (mgr.34547) 19 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:57.252 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:56 smithi204 bash[35809]: cluster 2024-02-10T09:30:55.902722+0000 mgr.x (mgr.34547) 19 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:57.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[44646]: Warning: Permanently added 'smithi204,172.21.15.204' (ECDSA) to the list of known hosts. 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.209089+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]: dispatch 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.209737+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]: dispatch 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.215463+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]': finished 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.216501+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.216883+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.220805+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:30:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.224864+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[39117]: audit 2024-02-10T09:30:57.422375+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.209089+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.209737+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.215463+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]': finished 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.216501+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.216883+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.220805+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.224864+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:30:58.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:57 smithi153 bash[35780]: audit 2024-02-10T09:30:57.422375+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.209089+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]: dispatch 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.209737+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]: dispatch 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.215463+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992", "who": "global"}]': finished 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.216501+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.216883+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.220805+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.224864+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:30:58.229 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:57 smithi204 bash[35809]: audit 2024-02-10T09:30:57.422375+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:30:59.230 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:30:59 smithi204 bash[35809]: cluster 2024-02-10T09:30:57.903089+0000 mgr.x (mgr.34547) 20 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:30:59 smithi153 bash[35780]: cluster 2024-02-10T09:30:57.903089+0000 mgr.x (mgr.34547) 20 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:30:59.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:30:59 smithi153 bash[39117]: cluster 2024-02-10T09:30:57.903089+0000 mgr.x (mgr.34547) 20 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:00.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.642206+0000 mon.b (mon.2) 73 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.642925+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.645671+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.945381+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.945834+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.946121+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.946401+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.946675+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.946938+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.947253+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.947520+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:00 smithi204 bash[35809]: audit 2024-02-10T09:30:59.947781+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.642206+0000 mon.b (mon.2) 73 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:00.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.642206+0000 mon.b (mon.2) 73 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.642925+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:00.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.642925+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:00.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.645671+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:00.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.945381+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.945834+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.946121+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.946401+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.946675+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.946938+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.947253+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.947520+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[35780]: audit 2024-02-10T09:30:59.947781+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.645671+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}}, \"last_daemon_update\": \"2024-02-10T09:30:59.641065Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.945381+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.945834+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.946121+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.946401+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.946675+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.946938+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.947253+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:00.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.947520+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:00.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:00 smithi153 bash[39117]: audit 2024-02-10T09:30:59.947781+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:01 smithi204 bash[35809]: cluster 2024-02-10T09:30:59.903510+0000 mgr.x (mgr.34547) 21 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:01 smithi204 bash[35809]: audit 2024-02-10T09:31:01.153686+0000 mon.b (mon.2) 83 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:01 smithi204 bash[35809]: audit 2024-02-10T09:31:01.154537+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:01 smithi204 bash[35809]: audit 2024-02-10T09:31:01.160374+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[35780]: cluster 2024-02-10T09:30:59.903510+0000 mgr.x (mgr.34547) 21 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:01.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[39117]: cluster 2024-02-10T09:30:59.903510+0000 mgr.x (mgr.34547) 21 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[35780]: audit 2024-02-10T09:31:01.153686+0000 mon.b (mon.2) 83 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:01.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[39117]: audit 2024-02-10T09:31:01.153686+0000 mon.b (mon.2) 83 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[35780]: audit 2024-02-10T09:31:01.154537+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:01.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[39117]: audit 2024-02-10T09:31:01.154537+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:01.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[39117]: audit 2024-02-10T09:31:01.160374+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:01 smithi153 bash[35780]: audit 2024-02-10T09:31:01.160374+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106115891, \"cpu_percentage\": \"0.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.151944Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435997900, \"cpu_percentage\": \"3.37%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152080Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47825551, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152184Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152254Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45696942, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152319Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46200258, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152382Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152458Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 30796677, \"cpu_percentage\": \"0.59%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152495Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34309406, \"cpu_percentage\": \"0.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:01.152554Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:01.152662Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:02.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.463607+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.464095+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.464371+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.464616+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.464840+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.465055+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.465281+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.465491+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.465750+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.466844+0000 mon.b (mon.2) 93 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.467223+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: audit 2024-02-10T09:31:01.472816+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:02.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:02 smithi204 bash[35809]: cephadm 2024-02-10T09:31:01.482754+0000 mgr.x (mgr.34547) 22 : cephadm [INF] Deploying daemon alertmanager.a on smithi153 2024-02-10T09:31:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.463607+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.464095+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.464371+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.464616+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.464840+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.465055+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.465281+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.465491+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.465750+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.466844+0000 mon.b (mon.2) 93 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.467223+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: audit 2024-02-10T09:31:01.472816+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[35780]: cephadm 2024-02-10T09:31:01.482754+0000 mgr.x (mgr.34547) 22 : cephadm [INF] Deploying daemon alertmanager.a on smithi153 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.463607+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.464095+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.464371+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.464616+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.464840+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.465055+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.465281+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.465491+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.465750+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.466844+0000 mon.b (mon.2) 93 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.467223+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: audit 2024-02-10T09:31:01.472816+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:02 smithi153 bash[39117]: cephadm 2024-02-10T09:31:01.482754+0000 mgr.x (mgr.34547) 22 : cephadm [INF] Deploying daemon alertmanager.a on smithi153 2024-02-10T09:31:03.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:03 smithi204 bash[35809]: cluster 2024-02-10T09:31:01.904340+0000 mgr.x (mgr.34547) 23 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:03.688 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:03 smithi153 bash[35780]: cluster 2024-02-10T09:31:01.904340+0000 mgr.x (mgr.34547) 23 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:03 smithi153 bash[39117]: cluster 2024-02-10T09:31:01.904340+0000 mgr.x (mgr.34547) 23 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:05.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:05 smithi153 bash[35780]: cluster 2024-02-10T09:31:03.904778+0000 mgr.x (mgr.34547) 24 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:05.492 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:05 smithi153 bash[39117]: cluster 2024-02-10T09:31:03.904778+0000 mgr.x (mgr.34547) 24 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:05.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:05 smithi204 bash[35809]: cluster 2024-02-10T09:31:03.904778+0000 mgr.x (mgr.34547) 24 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:07.510 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:07 smithi153 bash[39117]: cluster 2024-02-10T09:31:05.905508+0000 mgr.x (mgr.34547) 25 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:07.510 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:07 smithi153 bash[35780]: cluster 2024-02-10T09:31:05.905508+0000 mgr.x (mgr.34547) 25 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:07.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:07 smithi204 bash[35809]: cluster 2024-02-10T09:31:05.905508+0000 mgr.x (mgr.34547) 25 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:08.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 systemd[1]: Starting Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.963Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.963Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.964Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.153 port=9094 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.965Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.991Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.991Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.994Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:08.994Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[59236]: c05b963121bb36bed581250eabacf7644dce9b01e2e1c38b1a2fe1535bdcea5f 2024-02-10T09:31:09.250 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:09 smithi153 systemd[1]: Started Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:31:09.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:09 smithi204 bash[35809]: cluster 2024-02-10T09:31:07.906030+0000 mgr.x (mgr.34547) 26 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:09.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:09 smithi204 bash[35809]: audit 2024-02-10T09:31:09.111052+0000 mon.b (mon.2) 94 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:09.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:09 smithi204 bash[35809]: audit 2024-02-10T09:31:09.111876+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:09.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:09 smithi204 bash[35809]: audit 2024-02-10T09:31:09.114944+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:09.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:09 smithi204 bash[35809]: audit 2024-02-10T09:31:09.120583+0000 mon.b (mon.2) 95 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:31:09.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:09 smithi204 bash[35809]: audit 2024-02-10T09:31:09.121104+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:31:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[39117]: cluster 2024-02-10T09:31:07.906030+0000 mgr.x (mgr.34547) 26 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[35780]: cluster 2024-02-10T09:31:07.906030+0000 mgr.x (mgr.34547) 26 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.6 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:09.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[39117]: audit 2024-02-10T09:31:09.111052+0000 mon.b (mon.2) 94 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:09.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[35780]: audit 2024-02-10T09:31:09.111052+0000 mon.b (mon.2) 94 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:09.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[39117]: audit 2024-02-10T09:31:09.111876+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:09.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[39117]: audit 2024-02-10T09:31:09.114944+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:09.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[39117]: audit 2024-02-10T09:31:09.120583+0000 mon.b (mon.2) 95 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:31:09.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[39117]: audit 2024-02-10T09:31:09.121104+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:31:09.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[35780]: audit 2024-02-10T09:31:09.111876+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:09.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[35780]: audit 2024-02-10T09:31:09.114944+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 89642762, \"cpu_percentage\": \"1.33%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640322Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 301780172, \"cpu_percentage\": \"2.83%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640524Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640598Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43998248, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640664Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45067796, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640728Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42331013, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640789Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45875200, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640851Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640912Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 32925286, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:30:59.640949Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:09.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[35780]: audit 2024-02-10T09:31:09.120583+0000 mon.b (mon.2) 95 : audit [INF] from='mgr.34547 172.21.15.204:0/2319993333' entity='mgr.x' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:31:09.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:09 smithi153 bash[35780]: audit 2024-02-10T09:31:09.121104+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:31:10.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:10 smithi153 bash[36014]: ignoring --setuser ceph since I am not root 2024-02-10T09:31:10.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:10 smithi153 bash[36014]: ignoring --setgroup ceph since I am not root 2024-02-10T09:31:10.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:10 smithi204 bash[44646]: ignoring --setuser ceph since I am not root 2024-02-10T09:31:10.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:10 smithi204 bash[44646]: ignoring --setgroup ceph since I am not root 2024-02-10T09:31:10.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:10 smithi204 bash[44646]: debug 2024-02-10T09:31:10.160+0000 7ff8380a6700 1 -- 172.21.15.204:0/2006027493 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a39e18a4e0 con 0x55a39e160c00 2024-02-10T09:31:10.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:10 smithi204 bash[44646]: debug 2024-02-10T09:31:10.282+0000 7ff848284680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:31:10.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:10 smithi204 bash[44646]: debug 2024-02-10T09:31:10.363+0000 7ff848284680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:31:11.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:10 smithi204 bash[44646]: debug 2024-02-10T09:31:10.727+0000 7ff848284680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:31:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:11 smithi153 bash[35780]: audit 2024-02-10T09:31:10.120482+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T09:31:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:11 smithi153 bash[35780]: cluster 2024-02-10T09:31:10.120550+0000 mon.a (mon.0) 655 : cluster [DBG] mgrmap e21: x(active, since 52s), standbys: y 2024-02-10T09:31:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:11 smithi153 bash[39117]: audit 2024-02-10T09:31:10.120482+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T09:31:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:11 smithi153 bash[39117]: cluster 2024-02-10T09:31:10.120550+0000 mon.a (mon.0) 655 : cluster [DBG] mgrmap e21: x(active, since 52s), standbys: y 2024-02-10T09:31:11.440 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:10.966Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000150724s 2024-02-10T09:31:11.474 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:11 smithi204 bash[35809]: audit 2024-02-10T09:31:10.120482+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34547 ' entity='mgr.x' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T09:31:11.474 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:11 smithi204 bash[35809]: cluster 2024-02-10T09:31:10.120550+0000 mon.a (mon.0) 655 : cluster [DBG] mgrmap e21: x(active, since 52s), standbys: y 2024-02-10T09:31:11.474 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:11 smithi204 bash[44646]: debug 2024-02-10T09:31:11.471+0000 7ff848284680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:31:11.770 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:11 smithi204 bash[44646]: debug 2024-02-10T09:31:11.556+0000 7ff848284680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:31:12.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:11 smithi204 bash[44646]: debug 2024-02-10T09:31:11.769+0000 7ff848284680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:31:13.003 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:12 smithi204 bash[44646]: debug 2024-02-10T09:31:12.703+0000 7ff848284680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:31:13.003 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:12 smithi204 bash[44646]: debug 2024-02-10T09:31:12.831+0000 7ff848284680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:31:13.003 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:12 smithi204 bash[44646]: debug 2024-02-10T09:31:12.911+0000 7ff848284680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:31:13.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[44646]: debug 2024-02-10T09:31:13.001+0000 7ff848284680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:31:13.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[44646]: debug 2024-02-10T09:31:13.198+0000 7ff848284680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:31:13.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[44646]: debug 2024-02-10T09:31:13.477+0000 7ff848284680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:31:13.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[44646]: debug 2024-02-10T09:31:13.576+0000 7ff848284680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:31:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[39117]: cluster 2024-02-10T09:31:13.770070+0000 mon.a (mon.0) 656 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:31:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[39117]: cluster 2024-02-10T09:31:13.770165+0000 mon.a (mon.0) 657 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:31:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[39117]: audit 2024-02-10T09:31:13.771538+0000 mon.c (mon.1) 33 : audit [DBG] from='mgr.? 172.21.15.153:0/3250015070' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:31:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[39117]: audit 2024-02-10T09:31:13.773825+0000 mon.c (mon.1) 34 : audit [DBG] from='mgr.? 172.21.15.153:0/3250015070' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:31:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[35780]: cluster 2024-02-10T09:31:13.770070+0000 mon.a (mon.0) 656 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:31:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[35780]: cluster 2024-02-10T09:31:13.770165+0000 mon.a (mon.0) 657 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[35780]: audit 2024-02-10T09:31:13.771538+0000 mon.c (mon.1) 33 : audit [DBG] from='mgr.? 172.21.15.153:0/3250015070' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[35780]: audit 2024-02-10T09:31:13.773825+0000 mon.c (mon.1) 34 : audit [DBG] from='mgr.? 172.21.15.153:0/3250015070' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[36014]: [10/Feb/2024:09:31:13] ENGINE Bus STARTING 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[36014]: CherryPy Checker: 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[36014]: The Application mounted at '' has an empty config. 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[36014]: [10/Feb/2024:09:31:13] ENGINE Serving on http://:::9283 2024-02-10T09:31:14.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:13 smithi153 bash[36014]: [10/Feb/2024:09:31:13] ENGINE Bus STARTED 2024-02-10T09:31:14.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[35809]: cluster 2024-02-10T09:31:13.770070+0000 mon.a (mon.0) 656 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:31:14.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[35809]: cluster 2024-02-10T09:31:13.770165+0000 mon.a (mon.0) 657 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:31:14.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[35809]: audit 2024-02-10T09:31:13.771538+0000 mon.c (mon.1) 33 : audit [DBG] from='mgr.? 172.21.15.153:0/3250015070' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:31:14.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:13 smithi204 bash[35809]: audit 2024-02-10T09:31:13.773825+0000 mon.c (mon.1) 34 : audit [DBG] from='mgr.? 172.21.15.153:0/3250015070' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:31:14.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:14 smithi204 bash[44646]: debug 2024-02-10T09:31:14.306+0000 7ff848284680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:31:14.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:14 smithi204 bash[44646]: debug 2024-02-10T09:31:14.389+0000 7ff848284680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:31:14.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:14 smithi204 bash[44646]: debug 2024-02-10T09:31:14.484+0000 7ff848284680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:31:14.925 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:14 smithi204 bash[35809]: cluster 2024-02-10T09:31:13.825425+0000 mon.a (mon.0) 658 : cluster [DBG] mgrmap e22: x(active, since 56s), standbys: y 2024-02-10T09:31:14.926 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:14 smithi204 bash[44646]: debug 2024-02-10T09:31:14.662+0000 7ff848284680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:31:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:14 smithi153 bash[35780]: cluster 2024-02-10T09:31:13.825425+0000 mon.a (mon.0) 658 : cluster [DBG] mgrmap e22: x(active, since 56s), standbys: y 2024-02-10T09:31:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:14 smithi153 bash[39117]: cluster 2024-02-10T09:31:13.825425+0000 mon.a (mon.0) 658 : cluster [DBG] mgrmap e22: x(active, since 56s), standbys: y 2024-02-10T09:31:15.192 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:14 smithi204 bash[44646]: debug 2024-02-10T09:31:14.924+0000 7ff848284680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:31:15.192 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[44646]: debug 2024-02-10T09:31:15.056+0000 7ff848284680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:31:15.448 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[44646]: debug 2024-02-10T09:31:15.191+0000 7ff848284680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:31:15.448 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[44646]: debug 2024-02-10T09:31:15.362+0000 7ff848284680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:31:15.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[44646]: debug 2024-02-10T09:31:15.447+0000 7ff848284680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:31:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:15 smithi153 bash[35780]: cluster 2024-02-10T09:31:15.461925+0000 mon.a (mon.0) 659 : cluster [INF] Active manager daemon x restarted 2024-02-10T09:31:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:15 smithi153 bash[35780]: cluster 2024-02-10T09:31:15.462671+0000 mon.a (mon.0) 660 : cluster [INF] Activating manager daemon x 2024-02-10T09:31:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:15 smithi153 bash[35780]: cluster 2024-02-10T09:31:15.468788+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-10T09:31:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:15 smithi153 bash[39117]: cluster 2024-02-10T09:31:15.461925+0000 mon.a (mon.0) 659 : cluster [INF] Active manager daemon x restarted 2024-02-10T09:31:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:15 smithi153 bash[39117]: cluster 2024-02-10T09:31:15.462671+0000 mon.a (mon.0) 660 : cluster [INF] Activating manager daemon x 2024-02-10T09:31:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:15 smithi153 bash[39117]: cluster 2024-02-10T09:31:15.468788+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-10T09:31:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[35809]: cluster 2024-02-10T09:31:15.461925+0000 mon.a (mon.0) 659 : cluster [INF] Active manager daemon x restarted 2024-02-10T09:31:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[35809]: cluster 2024-02-10T09:31:15.462671+0000 mon.a (mon.0) 660 : cluster [INF] Activating manager daemon x 2024-02-10T09:31:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:15 smithi204 bash[35809]: cluster 2024-02-10T09:31:15.468788+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-10T09:31:17.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:16 smithi153 bash[35780]: cluster 2024-02-10T09:31:15.850560+0000 mon.a (mon.0) 662 : cluster [DBG] mgrmap e23: x(active, starting, since 0.388087s), standbys: y 2024-02-10T09:31:17.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:16 smithi153 bash[39117]: cluster 2024-02-10T09:31:15.850560+0000 mon.a (mon.0) 662 : cluster [DBG] mgrmap e23: x(active, starting, since 0.388087s), standbys: y 2024-02-10T09:31:17.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:16 smithi204 bash[35809]: cluster 2024-02-10T09:31:15.850560+0000 mon.a (mon.0) 662 : cluster [DBG] mgrmap e23: x(active, starting, since 0.388087s), standbys: y 2024-02-10T09:31:19.439 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:18.969Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003116172s 2024-02-10T09:31:22.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: [10/Feb/2024:09:31:22] ENGINE Bus STARTING 2024-02-10T09:31:22.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: Warning: Permanently added '172.21.15.153' (ECDSA) to the list of known hosts. 2024-02-10T09:31:22.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: Warning: Permanently added 'smithi204,172.21.15.204' (ECDSA) to the list of known hosts. 2024-02-10T09:31:22.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: CherryPy Checker: 2024-02-10T09:31:22.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: The Application mounted at '' has an empty config. 2024-02-10T09:31:22.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: [10/Feb/2024:09:31:22] ENGINE Serving on http://:::9283 2024-02-10T09:31:22.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[44646]: [10/Feb/2024:09:31:22] ENGINE Bus STARTED 2024-02-10T09:31:22.798 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[55964]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:31:22.798 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[55964]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:31:22.798 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[55964]: Error: 3 errors occurred while pulling: 2024-02-10T09:31:22.798 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[55964]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:31:22.798 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[55964]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:31:22.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.851787+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:31:22.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.852050+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:31:22.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.852259+0000 mon.b (mon.2) 98 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.853107+0000 mon.b (mon.2) 99 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.853346+0000 mon.b (mon.2) 100 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.853583+0000 mon.b (mon.2) 101 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.853855+0000 mon.b (mon.2) 102 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.854103+0000 mon.b (mon.2) 103 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.854358+0000 mon.b (mon.2) 104 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.854578+0000 mon.b (mon.2) 105 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.854822+0000 mon.b (mon.2) 106 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.855056+0000 mon.b (mon.2) 107 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.855304+0000 mon.b (mon.2) 108 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.856045+0000 mon.b (mon.2) 109 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.856286+0000 mon.b (mon.2) 110 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.856778+0000 mon.b (mon.2) 111 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.857500+0000 mon.b (mon.2) 112 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:31:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.858321+0000 mon.b (mon.2) 113 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.851787+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.852050+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.852259+0000 mon.b (mon.2) 98 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.853107+0000 mon.b (mon.2) 99 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.853346+0000 mon.b (mon.2) 100 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.853583+0000 mon.b (mon.2) 101 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.853855+0000 mon.b (mon.2) 102 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.854103+0000 mon.b (mon.2) 103 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:31:22.800 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.854358+0000 mon.b (mon.2) 104 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.854578+0000 mon.b (mon.2) 105 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.854822+0000 mon.b (mon.2) 106 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.855056+0000 mon.b (mon.2) 107 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.855304+0000 mon.b (mon.2) 108 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.856045+0000 mon.b (mon.2) 109 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.856286+0000 mon.b (mon.2) 110 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.856778+0000 mon.b (mon.2) 111 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.857500+0000 mon.b (mon.2) 112 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.858321+0000 mon.b (mon.2) 113 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.858686+0000 mon.b (mon.2) 114 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.859043+0000 mon.b (mon.2) 115 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.859399+0000 mon.b (mon.2) 116 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.859751+0000 mon.b (mon.2) 117 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.860095+0000 mon.b (mon.2) 118 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.860422+0000 mon.b (mon.2) 119 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.860726+0000 mon.b (mon.2) 120 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:31:22.801 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.861040+0000 mon.b (mon.2) 121 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.861378+0000 mon.b (mon.2) 122 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.861679+0000 mon.b (mon.2) 123 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.861986+0000 mon.b (mon.2) 124 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.862300+0000 mon.b (mon.2) 125 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.862607+0000 mon.b (mon.2) 126 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.862936+0000 mon.b (mon.2) 127 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.863256+0000 mon.b (mon.2) 128 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.863569+0000 mon.b (mon.2) 129 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.858686+0000 mon.b (mon.2) 114 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.859043+0000 mon.b (mon.2) 115 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.859399+0000 mon.b (mon.2) 116 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:31:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.859751+0000 mon.b (mon.2) 117 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.860095+0000 mon.b (mon.2) 118 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.860422+0000 mon.b (mon.2) 119 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.860726+0000 mon.b (mon.2) 120 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.861040+0000 mon.b (mon.2) 121 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.861378+0000 mon.b (mon.2) 122 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.861679+0000 mon.b (mon.2) 123 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.861986+0000 mon.b (mon.2) 124 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.862300+0000 mon.b (mon.2) 125 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.862607+0000 mon.b (mon.2) 126 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.862936+0000 mon.b (mon.2) 127 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.863256+0000 mon.b (mon.2) 128 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:31:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.863569+0000 mon.b (mon.2) 129 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.863876+0000 mon.b (mon.2) 130 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.864189+0000 mon.b (mon.2) 131 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.864507+0000 mon.b (mon.2) 132 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.864815+0000 mon.b (mon.2) 133 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.865119+0000 mon.b (mon.2) 134 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: cluster 2024-02-10T09:31:21.880338+0000 mon.a (mon.0) 663 : cluster [INF] Manager daemon x is now available 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.982951+0000 mon.b (mon.2) 135 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:21.984300+0000 mon.b (mon.2) 136 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.016159+0000 mon.b (mon.2) 137 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.016706+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.020449+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]': finished 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.033229+0000 mon.b (mon.2) 138 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.033686+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:22.804 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.048416+0000 mon.b (mon.2) 139 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.048845+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.219999+0000 mon.b (mon.2) 140 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[35780]: audit 2024-02-10T09:31:22.271355+0000 mon.b (mon.2) 141 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.863876+0000 mon.b (mon.2) 130 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.864189+0000 mon.b (mon.2) 131 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.864507+0000 mon.b (mon.2) 132 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.864815+0000 mon.b (mon.2) 133 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.865119+0000 mon.b (mon.2) 134 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: cluster 2024-02-10T09:31:21.880338+0000 mon.a (mon.0) 663 : cluster [INF] Manager daemon x is now available 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.982951+0000 mon.b (mon.2) 135 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:21.984300+0000 mon.b (mon.2) 136 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.016159+0000 mon.b (mon.2) 137 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]: dispatch 2024-02-10T09:31:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.016706+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]: dispatch 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.020449+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]': finished 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.033229+0000 mon.b (mon.2) 138 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.033686+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.048416+0000 mon.b (mon.2) 139 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.048845+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.219999+0000 mon.b (mon.2) 140 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[39117]: audit 2024-02-10T09:31:22.271355+0000 mon.b (mon.2) 141 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:23.023 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.851787+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:31:23.023 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.852050+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.852259+0000 mon.b (mon.2) 98 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.853107+0000 mon.b (mon.2) 99 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.853346+0000 mon.b (mon.2) 100 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.853583+0000 mon.b (mon.2) 101 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.853855+0000 mon.b (mon.2) 102 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.854103+0000 mon.b (mon.2) 103 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.854358+0000 mon.b (mon.2) 104 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.854578+0000 mon.b (mon.2) 105 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.854822+0000 mon.b (mon.2) 106 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.855056+0000 mon.b (mon.2) 107 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.855304+0000 mon.b (mon.2) 108 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.856045+0000 mon.b (mon.2) 109 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.856286+0000 mon.b (mon.2) 110 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.856778+0000 mon.b (mon.2) 111 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:31:23.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.857500+0000 mon.b (mon.2) 112 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.858321+0000 mon.b (mon.2) 113 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.858686+0000 mon.b (mon.2) 114 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.859043+0000 mon.b (mon.2) 115 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.859399+0000 mon.b (mon.2) 116 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.859751+0000 mon.b (mon.2) 117 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.860095+0000 mon.b (mon.2) 118 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.860422+0000 mon.b (mon.2) 119 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.860726+0000 mon.b (mon.2) 120 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.861040+0000 mon.b (mon.2) 121 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.861378+0000 mon.b (mon.2) 122 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.861679+0000 mon.b (mon.2) 123 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.861986+0000 mon.b (mon.2) 124 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.862300+0000 mon.b (mon.2) 125 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.862607+0000 mon.b (mon.2) 126 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:31:23.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.862936+0000 mon.b (mon.2) 127 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.863256+0000 mon.b (mon.2) 128 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.863569+0000 mon.b (mon.2) 129 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.863876+0000 mon.b (mon.2) 130 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.864189+0000 mon.b (mon.2) 131 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.864507+0000 mon.b (mon.2) 132 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.864815+0000 mon.b (mon.2) 133 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.865119+0000 mon.b (mon.2) 134 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: cluster 2024-02-10T09:31:21.880338+0000 mon.a (mon.0) 663 : cluster [INF] Manager daemon x is now available 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.982951+0000 mon.b (mon.2) 135 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:21.984300+0000 mon.b (mon.2) 136 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.016159+0000 mon.b (mon.2) 137 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.016706+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.020449+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{}"}]': finished 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.033229+0000 mon.b (mon.2) 138 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.033686+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:23.026 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.048416+0000 mon.b (mon.2) 139 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:23.027 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.048845+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:23.027 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.219999+0000 mon.b (mon.2) 140 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:23.027 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:22 smithi204 bash[35809]: audit 2024-02-10T09:31:22.271355+0000 mon.b (mon.2) 141 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:23.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 bash[55964]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:31:23.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=125/n/a 2024-02-10T09:31:23.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:22 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:31:23.293 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[45101]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:31:23.293 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[45101]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:31:23.293 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[45101]: Error: 3 errors occurred while pulling: 2024-02-10T09:31:23.293 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[45101]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:31:23.293 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[45101]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:31:23.293 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[45101]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:31:23.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=125/n/a 2024-02-10T09:31:23.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:23 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:31:24.074 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[35809]: cluster 2024-02-10T09:31:22.733589+0000 mon.a (mon.0) 668 : cluster [DBG] mgrmap e24: x(active, since 7s), standbys: y 2024-02-10T09:31:24.074 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:23 smithi204 bash[35809]: cluster 2024-02-10T09:31:22.737542+0000 mgr.x (mgr.34643) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:23 smithi153 bash[39117]: cluster 2024-02-10T09:31:22.733589+0000 mon.a (mon.0) 668 : cluster [DBG] mgrmap e24: x(active, since 7s), standbys: y 2024-02-10T09:31:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:23 smithi153 bash[39117]: cluster 2024-02-10T09:31:22.737542+0000 mgr.x (mgr.34643) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:23 smithi153 bash[35780]: cluster 2024-02-10T09:31:22.733589+0000 mon.a (mon.0) 668 : cluster [DBG] mgrmap e24: x(active, since 7s), standbys: y 2024-02-10T09:31:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:23 smithi153 bash[35780]: cluster 2024-02-10T09:31:22.737542+0000 mgr.x (mgr.34643) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:24 smithi204 bash[35809]: cluster 2024-02-10T09:31:23.743808+0000 mon.a (mon.0) 669 : cluster [DBG] mgrmap e25: x(active, since 8s), standbys: y 2024-02-10T09:31:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:24 smithi204 bash[35809]: cluster 2024-02-10T09:31:23.856193+0000 mgr.x (mgr.34643) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:24 smithi153 bash[35780]: cluster 2024-02-10T09:31:23.743808+0000 mon.a (mon.0) 669 : cluster [DBG] mgrmap e25: x(active, since 8s), standbys: y 2024-02-10T09:31:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:24 smithi153 bash[35780]: cluster 2024-02-10T09:31:23.856193+0000 mgr.x (mgr.34643) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:24 smithi153 bash[39117]: cluster 2024-02-10T09:31:23.743808+0000 mon.a (mon.0) 669 : cluster [DBG] mgrmap e25: x(active, since 8s), standbys: y 2024-02-10T09:31:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:24 smithi153 bash[39117]: cluster 2024-02-10T09:31:23.856193+0000 mgr.x (mgr.34643) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:26.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.338411+0000 mon.b (mon.2) 142 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.339103+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.342429+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.621132+0000 mon.b (mon.2) 143 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.621483+0000 mon.b (mon.2) 144 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.621752+0000 mon.b (mon.2) 145 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.621987+0000 mon.b (mon.2) 146 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.622241+0000 mon.b (mon.2) 147 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.622480+0000 mon.b (mon.2) 148 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.622709+0000 mon.b (mon.2) 149 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.622926+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:25 smithi204 bash[35809]: audit 2024-02-10T09:31:25.623165+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.338411+0000 mon.b (mon.2) 142 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:26.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.338411+0000 mon.b (mon.2) 142 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:26.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.339103+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:26.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.339103+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:26.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.342429+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:26.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.621132+0000 mon.b (mon.2) 143 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.093 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.342429+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.621132+0000 mon.b (mon.2) 143 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.621483+0000 mon.b (mon.2) 144 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.621752+0000 mon.b (mon.2) 145 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.621987+0000 mon.b (mon.2) 146 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.622241+0000 mon.b (mon.2) 147 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.622480+0000 mon.b (mon.2) 148 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.622709+0000 mon.b (mon.2) 149 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.622926+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[39117]: audit 2024-02-10T09:31:25.623165+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.621483+0000 mon.b (mon.2) 144 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.621752+0000 mon.b (mon.2) 145 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.621987+0000 mon.b (mon.2) 146 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.622241+0000 mon.b (mon.2) 147 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.622480+0000 mon.b (mon.2) 148 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.622709+0000 mon.b (mon.2) 149 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:26.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.622926+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:26.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:25 smithi153 bash[35780]: audit 2024-02-10T09:31:25.623165+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: cluster 2024-02-10T09:31:25.856555+0000 mgr.x (mgr.34643) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.046770+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.047359+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.050340+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.330985+0000 mon.b (mon.2) 153 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.331329+0000 mon.b (mon.2) 154 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.331573+0000 mon.b (mon.2) 155 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.331791+0000 mon.b (mon.2) 156 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.332012+0000 mon.b (mon.2) 157 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.332228+0000 mon.b (mon.2) 158 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.332443+0000 mon.b (mon.2) 159 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.332652+0000 mon.b (mon.2) 160 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.332872+0000 mon.b (mon.2) 161 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.333958+0000 mon.b (mon.2) 162 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:27.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.334608+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:27.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:26 smithi204 bash[35809]: audit 2024-02-10T09:31:26.339980+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:27.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: cluster 2024-02-10T09:31:25.856555+0000 mgr.x (mgr.34643) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:27.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: cluster 2024-02-10T09:31:25.856555+0000 mgr.x (mgr.34643) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:27.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.046770+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.046770+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.047359+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.050340+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.330985+0000 mon.b (mon.2) 153 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.331329+0000 mon.b (mon.2) 154 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.331573+0000 mon.b (mon.2) 155 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.331791+0000 mon.b (mon.2) 156 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.332012+0000 mon.b (mon.2) 157 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.332228+0000 mon.b (mon.2) 158 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.332443+0000 mon.b (mon.2) 159 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.332652+0000 mon.b (mon.2) 160 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.332872+0000 mon.b (mon.2) 161 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.333958+0000 mon.b (mon.2) 162 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.334608+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[35780]: audit 2024-02-10T09:31:26.339980+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.047359+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.050340+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:26.045666Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.330985+0000 mon.b (mon.2) 153 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.331329+0000 mon.b (mon.2) 154 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.331573+0000 mon.b (mon.2) 155 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.331791+0000 mon.b (mon.2) 156 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.332012+0000 mon.b (mon.2) 157 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.332228+0000 mon.b (mon.2) 158 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.332443+0000 mon.b (mon.2) 159 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.332652+0000 mon.b (mon.2) 160 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.332872+0000 mon.b (mon.2) 161 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.333958+0000 mon.b (mon.2) 162 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.334608+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:27.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:26 smithi153 bash[39117]: audit 2024-02-10T09:31:26.339980+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:27 smithi204 bash[35809]: cephadm 2024-02-10T09:31:26.354263+0000 mgr.x (mgr.34643) 5 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:31:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:27 smithi204 bash[35809]: cluster 2024-02-10T09:31:26.747896+0000 mon.a (mon.0) 676 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:31:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:27 smithi153 bash[35780]: cephadm 2024-02-10T09:31:26.354263+0000 mgr.x (mgr.34643) 5 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:31:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:27 smithi153 bash[35780]: cluster 2024-02-10T09:31:26.747896+0000 mon.a (mon.0) 676 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:31:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:27 smithi153 bash[39117]: cephadm 2024-02-10T09:31:26.354263+0000 mgr.x (mgr.34643) 5 : cephadm [INF] Deploying daemon prometheus.a on smithi204 2024-02-10T09:31:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:27 smithi153 bash[39117]: cluster 2024-02-10T09:31:26.747896+0000 mon.a (mon.0) 676 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:31:30.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:29 smithi204 bash[35809]: cluster 2024-02-10T09:31:28.334567+0000 mgr.x (mgr.34643) 6 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:29 smithi153 bash[35780]: cluster 2024-02-10T09:31:28.334567+0000 mgr.x (mgr.34643) 6 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:29 smithi153 bash[39117]: cluster 2024-02-10T09:31:28.334567+0000 mgr.x (mgr.34643) 6 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:31 smithi204 bash[35809]: cluster 2024-02-10T09:31:30.335121+0000 mgr.x (mgr.34643) 7 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:32.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:31 smithi153 bash[35780]: cluster 2024-02-10T09:31:30.335121+0000 mgr.x (mgr.34643) 7 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:31 smithi153 bash[39117]: cluster 2024-02-10T09:31:30.335121+0000 mgr.x (mgr.34643) 7 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:33.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:32 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:31:33.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:32 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:31:33.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:32 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:31:33.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:32 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:31:33.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:33 smithi153 bash[59946]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:31:33.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:31:33 smithi153 bash[59946]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:31:33.775 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:31:33.776 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:31:33.776 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:31:33.776 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:31:33.776 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 podman[48144]: Error: no container with ID or name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T09:31:33.776 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 systemd[1]: Started Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:31:33.776 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 bash[48159]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:31:33.777 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:31:33 smithi204 bash[48159]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:31:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:33 smithi204 bash[35809]: cluster 2024-02-10T09:31:32.335545+0000 mgr.x (mgr.34643) 8 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:33 smithi153 bash[35780]: cluster 2024-02-10T09:31:32.335545+0000 mgr.x (mgr.34643) 8 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:33 smithi153 bash[39117]: cluster 2024-02-10T09:31:32.335545+0000 mgr.x (mgr.34643) 8 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:36.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:35 smithi204 bash[35809]: cluster 2024-02-10T09:31:34.336258+0000 mgr.x (mgr.34643) 9 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:35 smithi153 bash[35780]: cluster 2024-02-10T09:31:34.336258+0000 mgr.x (mgr.34643) 9 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:35 smithi153 bash[39117]: cluster 2024-02-10T09:31:34.336258+0000 mgr.x (mgr.34643) 9 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:37 smithi204 bash[35809]: cluster 2024-02-10T09:31:36.336818+0000 mgr.x (mgr.34643) 10 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:37 smithi204 bash[35809]: audit 2024-02-10T09:31:37.024675+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]: dispatch 2024-02-10T09:31:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:37 smithi204 bash[35809]: audit 2024-02-10T09:31:37.025238+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]: dispatch 2024-02-10T09:31:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:37 smithi204 bash[35809]: audit 2024-02-10T09:31:37.028321+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]': finished 2024-02-10T09:31:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[39117]: cluster 2024-02-10T09:31:36.336818+0000 mgr.x (mgr.34643) 10 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[39117]: audit 2024-02-10T09:31:37.024675+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]: dispatch 2024-02-10T09:31:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[39117]: audit 2024-02-10T09:31:37.025238+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]: dispatch 2024-02-10T09:31:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[39117]: audit 2024-02-10T09:31:37.028321+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]': finished 2024-02-10T09:31:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[35780]: cluster 2024-02-10T09:31:36.336818+0000 mgr.x (mgr.34643) 10 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:38.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[35780]: audit 2024-02-10T09:31:37.024675+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]: dispatch 2024-02-10T09:31:38.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[35780]: audit 2024-02-10T09:31:37.025238+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]: dispatch 2024-02-10T09:31:38.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:37 smithi153 bash[35780]: audit 2024-02-10T09:31:37.028321+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": true}}"}]': finished 2024-02-10T09:31:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:39 smithi204 bash[35809]: cluster 2024-02-10T09:31:38.337546+0000 mgr.x (mgr.34643) 11 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:39 smithi153 bash[35780]: cluster 2024-02-10T09:31:38.337546+0000 mgr.x (mgr.34643) 11 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:39 smithi153 bash[39117]: cluster 2024-02-10T09:31:38.337546+0000 mgr.x (mgr.34643) 11 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:41 smithi204 bash[35809]: cluster 2024-02-10T09:31:40.337921+0000 mgr.x (mgr.34643) 12 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:41 smithi153 bash[35780]: cluster 2024-02-10T09:31:40.337921+0000 mgr.x (mgr.34643) 12 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:41 smithi153 bash[39117]: cluster 2024-02-10T09:31:40.337921+0000 mgr.x (mgr.34643) 12 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:43 smithi204 bash[35809]: cluster 2024-02-10T09:31:42.338307+0000 mgr.x (mgr.34643) 13 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:43 smithi153 bash[35780]: cluster 2024-02-10T09:31:42.338307+0000 mgr.x (mgr.34643) 13 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:43 smithi153 bash[39117]: cluster 2024-02-10T09:31:42.338307+0000 mgr.x (mgr.34643) 13 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:45 smithi204 bash[35809]: cluster 2024-02-10T09:31:44.338919+0000 mgr.x (mgr.34643) 14 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:45 smithi153 bash[35780]: cluster 2024-02-10T09:31:44.338919+0000 mgr.x (mgr.34643) 14 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:45 smithi153 bash[39117]: cluster 2024-02-10T09:31:44.338919+0000 mgr.x (mgr.34643) 14 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:47.988 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:47 smithi204 bash[35809]: cluster 2024-02-10T09:31:46.339343+0000 mgr.x (mgr.34643) 15 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:47.988 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:47 smithi204 systemd[1]: Starting Ceph prometheus.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:31:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:47 smithi153 bash[35780]: cluster 2024-02-10T09:31:46.339343+0000 mgr.x (mgr.34643) 15 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:47 smithi153 bash[39117]: cluster 2024-02-10T09:31:46.339343+0000 mgr.x (mgr.34643) 15 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:48.581 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.178Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.178Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.178Z caller=main.go:518 level=info host_details="(Linux 4.18.0-536.el8.x86_64 #1 SMP Thu Jan 18 15:15:29 UTC 2024 x86_64 smithi204 (none))" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.178Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.178Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.180Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.180Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.181Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.183Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.183Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.635µs 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.183Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.183Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-10T09:31:48.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.183Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=36.866µs wal_replay_duration=283.034µs total_replay_duration=338.218µs 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.184Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.184Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.184Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.200Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.474967ms db_storage=515ns remote_storage=1.224µs web_handler=740ns query_engine=604ns scrape=439.965µs scrape_sd=29.307µs notify=25.889µs notify_sd=9.983µs rules=14.751628ms 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:31:48.200Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 bash[48579]: ac1f43abcfcc1202199d565bce0568037f17f39ecc15749a9b29becfdb6335e8 2024-02-10T09:31:48.583 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:31:48 smithi204 systemd[1]: Started Ceph prometheus.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:31:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:48 smithi204 bash[35809]: audit 2024-02-10T09:31:48.296387+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:49.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:48 smithi204 bash[35809]: audit 2024-02-10T09:31:48.297181+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:49.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:48 smithi204 bash[35809]: audit 2024-02-10T09:31:48.301040+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:49.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:48 smithi204 bash[35809]: audit 2024-02-10T09:31:48.302683+0000 mon.b (mon.2) 165 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:49.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:48 smithi204 bash[35809]: audit 2024-02-10T09:31:48.304007+0000 mon.b (mon.2) 166 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:49.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[35780]: audit 2024-02-10T09:31:48.296387+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:49.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[39117]: audit 2024-02-10T09:31:48.296387+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:49.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[35780]: audit 2024-02-10T09:31:48.297181+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:49.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[35780]: audit 2024-02-10T09:31:48.301040+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:49.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[35780]: audit 2024-02-10T09:31:48.302683+0000 mon.b (mon.2) 165 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:49.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[35780]: audit 2024-02-10T09:31:48.304007+0000 mon.b (mon.2) 166 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:49.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[39117]: audit 2024-02-10T09:31:48.297181+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:49.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[39117]: audit 2024-02-10T09:31:48.301040+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.044941Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 426980147, \"cpu_percentage\": \"5.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045078Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47993323, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045173Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48056238, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045240Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45791313, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045304Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46525317, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045367Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045445Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.57%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045482Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34655436, \"cpu_percentage\": \"0.24%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:26.045541Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9095]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:49.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[39117]: audit 2024-02-10T09:31:48.302683+0000 mon.b (mon.2) 165 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:31:49.093 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:48 smithi153 bash[39117]: audit 2024-02-10T09:31:48.304007+0000 mon.b (mon.2) 166 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:31:50.036 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:49 smithi204 bash[35809]: cluster 2024-02-10T09:31:48.339989+0000 mgr.x (mgr.34643) 16 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:49 smithi153 bash[35780]: cluster 2024-02-10T09:31:48.339989+0000 mgr.x (mgr.34643) 16 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:49 smithi153 bash[39117]: cluster 2024-02-10T09:31:48.339989+0000 mgr.x (mgr.34643) 16 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:51 smithi204 bash[35809]: cluster 2024-02-10T09:31:50.340375+0000 mgr.x (mgr.34643) 17 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:51 smithi153 bash[35780]: cluster 2024-02-10T09:31:50.340375+0000 mgr.x (mgr.34643) 17 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:51 smithi153 bash[39117]: cluster 2024-02-10T09:31:50.340375+0000 mgr.x (mgr.34643) 17 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.028220+0000 mon.b (mon.2) 167 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:31:53.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.028761+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:31:53.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.032079+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:31:53.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.712789+0000 mon.b (mon.2) 168 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:53.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.713808+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.717636+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.719405+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.720292+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.723088+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.726144+0000 mon.b (mon.2) 170 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.726681+0000 mon.b (mon.2) 171 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.727182+0000 mon.b (mon.2) 172 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:31:53.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:52 smithi204 bash[35809]: audit 2024-02-10T09:31:52.728000+0000 mon.b (mon.2) 173 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:31:53.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.028220+0000 mon.b (mon.2) 167 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:31:53.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.028761+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:31:53.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.028220+0000 mon.b (mon.2) 167 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:31:53.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.028761+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:31:53.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.032079+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:31:53.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.032079+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707556818.7739785, \"finished_at\": 1707557482.016496, \"add_to_ceph_s:\": 1707556815.599134}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707556847.0407863, \"finished_at\": 1707557482.0165048, \"add_to_ceph_s:\": 1707556834.5899186}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707556865.7608364, \"finished_at\": 1707557482.0165062, \"add_to_ceph_s:\": 1707556853.9054804}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707556879.842647, \"finished_at\": 1707557482.0165074, \"add_to_ceph_s:\": 1707556871.8922586}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557486.3420982, \"finished_at\": 1707557508.3018928, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:31:53.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.712789+0000 mon.b (mon.2) 168 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:53.093 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.712789+0000 mon.b (mon.2) 168 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:53.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.713808+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:53.093 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.713808+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:53.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.717636+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:53.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.717636+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:09.984528Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:53.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.719405+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:53.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.720292+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:53.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.723088+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:53.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.726144+0000 mon.b (mon.2) 170 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.726681+0000 mon.b (mon.2) 171 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.727182+0000 mon.b (mon.2) 172 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[35780]: audit 2024-02-10T09:31:52.728000+0000 mon.b (mon.2) 173 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.719405+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.720292+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.723088+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.726144+0000 mon.b (mon.2) 170 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.726681+0000 mon.b (mon.2) 171 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.727182+0000 mon.b (mon.2) 172 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:31:53.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:52 smithi153 bash[39117]: audit 2024-02-10T09:31:52.728000+0000 mon.b (mon.2) 173 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:31:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[35780]: cluster 2024-02-10T09:31:52.340737+0000 mgr.x (mgr.34643) 18 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[35780]: cluster 2024-02-10T09:31:52.718656+0000 mgr.x (mgr.34643) 19 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[35780]: cluster 2024-02-10T09:31:52.718826+0000 mgr.x (mgr.34643) 20 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[35780]: cephadm 2024-02-10T09:31:52.725893+0000 mgr.x (mgr.34643) 21 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-10T09:31:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[35780]: cephadm 2024-02-10T09:31:52.727669+0000 mgr.x (mgr.34643) 22 : cephadm [INF] Reconfiguring daemon mon.a on smithi153 2024-02-10T09:31:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[35780]: cluster 2024-02-10T09:31:53.029882+0000 mon.a (mon.0) 687 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:31:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[39117]: cluster 2024-02-10T09:31:52.340737+0000 mgr.x (mgr.34643) 18 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[39117]: cluster 2024-02-10T09:31:52.718656+0000 mgr.x (mgr.34643) 19 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[39117]: cluster 2024-02-10T09:31:52.718826+0000 mgr.x (mgr.34643) 20 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[39117]: cephadm 2024-02-10T09:31:52.725893+0000 mgr.x (mgr.34643) 21 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-10T09:31:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[39117]: cephadm 2024-02-10T09:31:52.727669+0000 mgr.x (mgr.34643) 22 : cephadm [INF] Reconfiguring daemon mon.a on smithi153 2024-02-10T09:31:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:53 smithi153 bash[39117]: cluster 2024-02-10T09:31:53.029882+0000 mon.a (mon.0) 687 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[35809]: cluster 2024-02-10T09:31:52.340737+0000 mgr.x (mgr.34643) 18 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[35809]: cluster 2024-02-10T09:31:52.718656+0000 mgr.x (mgr.34643) 19 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[35809]: cluster 2024-02-10T09:31:52.718826+0000 mgr.x (mgr.34643) 20 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[35809]: cephadm 2024-02-10T09:31:52.725893+0000 mgr.x (mgr.34643) 21 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[35809]: cephadm 2024-02-10T09:31:52.727669+0000 mgr.x (mgr.34643) 22 : cephadm [INF] Reconfiguring daemon mon.a on smithi153 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[35809]: cluster 2024-02-10T09:31:53.029882+0000 mon.a (mon.0) 687 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:31:54.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:31:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:31:53] "GET /metrics HTTP/1.1" 200 153183 "" "Prometheus/2.33.4" 2024-02-10T09:31:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:31:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:31:55.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.165596+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:55.134 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.165596+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:55.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.166351+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:55.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.170382+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:55.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.171545+0000 mon.b (mon.2) 175 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:31:55.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.172276+0000 mon.b (mon.2) 176 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:31:55.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.172925+0000 mon.b (mon.2) 177 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[35780]: audit 2024-02-10T09:31:54.173981+0000 mon.b (mon.2) 178 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.166351+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.170382+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.171545+0000 mon.b (mon.2) 175 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.172276+0000 mon.b (mon.2) 176 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.172925+0000 mon.b (mon.2) 177 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:31:55.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:54 smithi153 bash[39117]: audit 2024-02-10T09:31:54.173981+0000 mon.b (mon.2) 178 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:31:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.165596+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.166351+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:31:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.170382+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:17:59.239924Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:31:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.171545+0000 mon.b (mon.2) 175 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:31:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.172276+0000 mon.b (mon.2) 176 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:31:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.172925+0000 mon.b (mon.2) 177 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:31:55.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:54 smithi204 bash[35809]: audit 2024-02-10T09:31:54.173981+0000 mon.b (mon.2) 178 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:31:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[35780]: cephadm 2024-02-10T09:31:54.171251+0000 mgr.x (mgr.34643) 23 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T09:31:56.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[35780]: cephadm 2024-02-10T09:31:54.173553+0000 mgr.x (mgr.34643) 24 : cephadm [INF] Reconfiguring daemon mon.c on smithi153 2024-02-10T09:31:56.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[35780]: cluster 2024-02-10T09:31:54.719733+0000 mgr.x (mgr.34643) 25 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[39117]: cephadm 2024-02-10T09:31:54.171251+0000 mgr.x (mgr.34643) 23 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T09:31:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[39117]: cephadm 2024-02-10T09:31:54.173553+0000 mgr.x (mgr.34643) 24 : cephadm [INF] Reconfiguring daemon mon.c on smithi153 2024-02-10T09:31:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[39117]: cluster 2024-02-10T09:31:54.719733+0000 mgr.x (mgr.34643) 25 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:56.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[35780]: audit 2024-02-10T09:31:55.468225+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:31:56.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[39117]: audit 2024-02-10T09:31:55.468225+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:31:56.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[35780]: audit 2024-02-10T09:31:55.469142+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:31:56.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[35780]: audit 2024-02-10T09:31:55.472599+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:31:56.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[39117]: audit 2024-02-10T09:31:55.469142+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:31:56.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:56 smithi153 bash[39117]: audit 2024-02-10T09:31:55.472599+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:31:56.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:56 smithi204 bash[35809]: cephadm 2024-02-10T09:31:54.171251+0000 mgr.x (mgr.34643) 23 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T09:31:56.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:56 smithi204 bash[35809]: cephadm 2024-02-10T09:31:54.173553+0000 mgr.x (mgr.34643) 24 : cephadm [INF] Reconfiguring daemon mon.c on smithi153 2024-02-10T09:31:56.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:56 smithi204 bash[35809]: cluster 2024-02-10T09:31:54.719733+0000 mgr.x (mgr.34643) 25 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:56.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:56 smithi204 bash[35809]: audit 2024-02-10T09:31:55.468225+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:31:56.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:56 smithi204 bash[35809]: audit 2024-02-10T09:31:55.469142+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:31:56.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:56 smithi204 bash[35809]: audit 2024-02-10T09:31:55.472599+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"c05b963121bb\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 17280532, \"cpu_percentage\": \"0.73%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:31:25.337071Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:08.435207Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:01.482642Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:25.337191Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:31:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:57 smithi153 bash[35780]: cephadm 2024-02-10T09:31:55.474454+0000 mgr.x (mgr.34643) 26 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T09:31:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:57 smithi153 bash[35780]: cephadm 2024-02-10T09:31:55.481288+0000 mgr.x (mgr.34643) 27 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi153 2024-02-10T09:31:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:57 smithi153 bash[35780]: cluster 2024-02-10T09:31:56.720246+0000 mgr.x (mgr.34643) 28 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:57 smithi153 bash[39117]: cephadm 2024-02-10T09:31:55.474454+0000 mgr.x (mgr.34643) 26 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T09:31:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:57 smithi153 bash[39117]: cephadm 2024-02-10T09:31:55.481288+0000 mgr.x (mgr.34643) 27 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi153 2024-02-10T09:31:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:57 smithi153 bash[39117]: cluster 2024-02-10T09:31:56.720246+0000 mgr.x (mgr.34643) 28 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:57.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:57 smithi204 bash[35809]: cephadm 2024-02-10T09:31:55.474454+0000 mgr.x (mgr.34643) 26 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T09:31:57.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:57 smithi204 bash[35809]: cephadm 2024-02-10T09:31:55.481288+0000 mgr.x (mgr.34643) 27 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi153 2024-02-10T09:31:57.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:57 smithi204 bash[35809]: cluster 2024-02-10T09:31:56.720246+0000 mgr.x (mgr.34643) 28 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:31:58.771 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:58 smithi153 systemd[1]: Stopping Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:31:58.771 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:58 smithi153 bash[60301]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T09:31:59.085 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[59251]: level=info ts=2024-02-10T09:31:58.770Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T09:31:59.439 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:59 smithi153 bash[60339]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T09:32:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:31:59 smithi204 bash[35809]: cluster 2024-02-10T09:31:58.721097+0000 mgr.x (mgr.34643) 29 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:00.138 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:31:59 smithi153 bash[35780]: cluster 2024-02-10T09:31:58.721097+0000 mgr.x (mgr.34643) 29 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:00.139 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:31:59 smithi153 bash[39117]: cluster 2024-02-10T09:31:58.721097+0000 mgr.x (mgr.34643) 29 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:00.139 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:59 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-10T09:32:00.139 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:59 smithi153 systemd[1]: Stopped Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:32:00.139 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:31:59 smithi153 systemd[1]: Starting Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.296Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.296Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.297Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.153 port=9094 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.299Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.324Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.324Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.327Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:00.327Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[60462]: a318f98f94b4654da76a749c0b4afab0fe7b9134b87d5a4d664f6bda26733ec4 2024-02-10T09:32:00.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:00 smithi153 systemd[1]: Started Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:32:01.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.412157+0000 mon.b (mon.2) 180 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:32:01.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.413103+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:32:01.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.416308+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:32:01.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.418043+0000 mon.b (mon.2) 181 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:32:01.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.418833+0000 mon.b (mon.2) 182 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:32:01.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.419571+0000 mon.b (mon.2) 183 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:32:01.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:00 smithi204 bash[35809]: audit 2024-02-10T09:32:00.420705+0000 mon.b (mon.2) 184 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:32:01.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.412157+0000 mon.b (mon.2) 180 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:32:01.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.412157+0000 mon.b (mon.2) 180 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:32:01.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.413103+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:32:01.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.416308+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.418043+0000 mon.b (mon.2) 181 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.418833+0000 mon.b (mon.2) 182 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.419571+0000 mon.b (mon.2) 183 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[39117]: audit 2024-02-10T09:32:00.420705+0000 mon.b (mon.2) 184 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.413103+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.416308+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 97255424, \"cpu_percentage\": \"1.34%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336444Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 301885030, \"cpu_percentage\": \"3.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336603Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 107688755, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336676Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44155535, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336738Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44061163, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336798Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 42425384, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336862Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46053457, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336920Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.336977Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 33355202, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:25.337014Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [9093, 9094]}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:32:01.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.418043+0000 mon.b (mon.2) 181 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:32:01.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.418833+0000 mon.b (mon.2) 182 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:32:01.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.419571+0000 mon.b (mon.2) 183 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:32:01.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:00 smithi153 bash[35780]: audit 2024-02-10T09:32:00.420705+0000 mon.b (mon.2) 184 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:32:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:01 smithi204 bash[35809]: cephadm 2024-02-10T09:32:00.417723+0000 mgr.x (mgr.34643) 30 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T09:32:02.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:01 smithi204 bash[35809]: cephadm 2024-02-10T09:32:00.420197+0000 mgr.x (mgr.34643) 31 : cephadm [INF] Reconfiguring daemon mon.b on smithi204 2024-02-10T09:32:02.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:01 smithi204 bash[35809]: cluster 2024-02-10T09:32:00.721502+0000 mgr.x (mgr.34643) 32 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:02.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[44646]: [10/Feb/2024:09:32:02] ENGINE Bus STOPPING 2024-02-10T09:32:02.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[44646]: [10/Feb/2024:09:32:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:32:02.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[44646]: [10/Feb/2024:09:32:02] ENGINE Bus STOPPED 2024-02-10T09:32:02.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[44646]: [10/Feb/2024:09:32:02] ENGINE Bus STARTING 2024-02-10T09:32:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:01 smithi153 bash[39117]: cephadm 2024-02-10T09:32:00.417723+0000 mgr.x (mgr.34643) 30 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T09:32:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:01 smithi153 bash[39117]: cephadm 2024-02-10T09:32:00.420197+0000 mgr.x (mgr.34643) 31 : cephadm [INF] Reconfiguring daemon mon.b on smithi204 2024-02-10T09:32:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:01 smithi153 bash[39117]: cluster 2024-02-10T09:32:00.721502+0000 mgr.x (mgr.34643) 32 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:01 smithi153 bash[35780]: cephadm 2024-02-10T09:32:00.417723+0000 mgr.x (mgr.34643) 30 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T09:32:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:01 smithi153 bash[35780]: cephadm 2024-02-10T09:32:00.420197+0000 mgr.x (mgr.34643) 31 : cephadm [INF] Reconfiguring daemon mon.b on smithi204 2024-02-10T09:32:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:01 smithi153 bash[35780]: cluster 2024-02-10T09:32:00.721502+0000 mgr.x (mgr.34643) 32 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:02.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[44646]: [10/Feb/2024:09:32:02] ENGINE Serving on http://:::9283 2024-02-10T09:32:02.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[44646]: [10/Feb/2024:09:32:02] ENGINE Bus STARTED 2024-02-10T09:32:02.688 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:02.300Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000693987s 2024-02-10T09:32:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:01.908728+0000 mon.b (mon.2) 185 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:32:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:01.909564+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:32:03.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:01.913040+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: cephadm 2024-02-10T09:32:01.914070+0000 mgr.x (mgr.34643) 33 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.000727+0000 mon.b (mon.2) 186 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.001176+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.004176+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.006698+0000 mon.b (mon.2) 187 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.007259+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.009852+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.013628+0000 mon.b (mon.2) 188 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.014082+0000 mgr.x (mgr.34643) 34 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.014341+0000 mon.b (mon.2) 189 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.014950+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]: dispatch 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: audit 2024-02-10T09:32:02.018464+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]': finished 2024-02-10T09:32:03.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:02 smithi204 bash[35809]: cephadm 2024-02-10T09:32:02.027405+0000 mgr.x (mgr.34643) 35 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:32:03.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:01.908728+0000 mon.b (mon.2) 185 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:32:03.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:01.908728+0000 mon.b (mon.2) 185 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:32:03.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:01.909564+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:32:03.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:01.913040+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:32:03.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: cephadm 2024-02-10T09:32:01.914070+0000 mgr.x (mgr.34643) 33 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:32:03.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.000727+0000 mon.b (mon.2) 186 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:32:03.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.001176+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.004176+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.006698+0000 mon.b (mon.2) 187 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.007259+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.009852+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.013628+0000 mon.b (mon.2) 188 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.014082+0000 mgr.x (mgr.34643) 34 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.014341+0000 mon.b (mon.2) 189 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.014950+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]: dispatch 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: audit 2024-02-10T09:32:02.018464+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]': finished 2024-02-10T09:32:03.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[39117]: cephadm 2024-02-10T09:32:02.027405+0000 mgr.x (mgr.34643) 35 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:32:03.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:01.909564+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:32:03.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:01.913040+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 106220748, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710842Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 438199910, \"cpu_percentage\": \"5.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.710983Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48087695, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711051Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48276439, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711113Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45917143, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711199Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46703575, \"cpu_percentage\": \"0.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711261Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711323Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"a148afe5b831\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 31142707, \"cpu_percentage\": \"0.55%\", \"version\": \"6.6.2\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711361Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:29:00.545771Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 34770780, \"cpu_percentage\": \"0.22%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:31:52.711421Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 33166458, \"cpu_percentage\": \"25.93%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:31:52.711512Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:31:52.711623Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:32:03.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: cephadm 2024-02-10T09:32:01.914070+0000 mgr.x (mgr.34643) 33 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:32:03.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.000727+0000 mon.b (mon.2) 186 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:32:03.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.001176+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.004176+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_crt","val":"-----BEGIN CERTIFICATE-----\nMIICyjCCAbICEQDRXpcY1n9EVoxLvkFspqDGMA0GCSqGSIb3DQEBDQUAMCMxDTAL\nBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRoaTIwNDAeFw0yNDAyMTAwOTMyMDFa\nFw0zNDAyMDcwOTMyMDFaMCMxDTALBgNVBAoMBENlcGgxEjAQBgNVBAMMCXNtaXRo\naTIwNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOANqhDz8TkBK2JD\n/jWOSdbIdCAYfxn7rnPpYf1gQlcl98kf74cTI52p1RRZl3p/NSZ7gIcl3PaQj62E\nk0AuL1jBAtPS0wJkzHqlPGOZ4FZzOg8WEVVIYwegwxVs/gwf7tXaDADfqUlyhHOm\nyFujPYwwapkg/Pui+4uX8UwX5+WYQD+XrMYphr63KnTYH64duNAtDy5s3OC0am+G\n8NnYgOAu+Mb9LshMzzQupIg03ERrW9CzEC25Peu3VyHbkKnzPu0ij+meOv11284s\n+T6byotHuey1g9sn2mYgCc5X9TU13lhP/8CFbJq0em4SvUECaK7vd6bHDm/Ozr+r\n7utqfmkCAwEAATANBgkqhkiG9w0BAQ0FAAOCAQEAbQCxUf6CVXGThDu7TMsfN+eI\np9xdyysYv8I4rDlXfUxywfZO5iydvZ1waQDqkSPs4tMfzhAWuE59gDGm6djmM1zc\nwhCdd8wrngY/E9RVZdKDDoa7SjiFoChI3LNEXgJSAUVkwUNqYn9ZtSMFBMe4XKUU\nelb3+up1QDc1/RtpgAm4zuudTgfCwolEba4DayAaEdDOwXhMnzU+1xYgrBltArmE\nJbDxCwHAFJfS2r/J/xuqsCzqYliwHe35S/YKU3ie7f5LyzxTzCNKndNyNHn3tHbU\n24996EnTCizOg307wdwcwcp82BQGy1CyqWpLNOD5xIuOnHA/PzlinTcj/cqGhQ==\n-----END CERTIFICATE-----\n"}]': finished 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.006698+0000 mon.b (mon.2) 187 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.007259+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.009852+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/smithi204/grafana_key","val":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgDaoQ8/E5ASti\nQ/41jknWyHQgGH8Z+65z6WH9YEJXJffJH++HEyOdqdUUWZd6fzUme4CHJdz2kI+t\nhJNALi9YwQLT0tMCZMx6pTxjmeBWczoPFhFVSGMHoMMVbP4MH+7V2gwA36lJcoRz\npshboz2MMGqZIPz7ovuLl/FMF+flmEA/l6zGKYa+typ02B+uHbjQLQ8ubNzgtGpv\nhvDZ2IDgLvjG/S7ITM80LqSINNxEa1vQsxAtuT3rt1ch25Cp8z7tIo/pnjr9ddvO\nLPk+m8qLR7nstYPbJ9pmIAnOV/U1Nd5YT//AhWyatHpuEr1BAmiu73emxw5vzs6/\nq+7ran5pAgMBAAECggEAI8prAVxGi2mhaYgdAqchKntI+ZuQKjFnM8LzUUtgLZW2\nGF3OoJAxH01y6VZKeVn6vpP5+xzUSFrdaMWF1DnMfGwQrI+//2DkvS7KD8sT6Btu\nCdznle0DTMOkhjar+nnbaDP1uOoBm+b8O5FH7kOMYGmXjbCUWQo3DpyOAi7Lb4Rv\nMtF0Vb2Aj1JjM+ODA5HJoQ2gD0o+ebi1+u7VbpEF+qIpC4X0wLeI4AK5zXe5+mN5\nao5tNH6vxITox/F9IVOtB3uySlEEQZ6Gq/JAA6EJuxLCJ7l3Syh+izdfU76F0hsJ\nT8F13IOOgzu6DtYw3WA6vcaXMb75FKe32G5LKipbkQKBgQDwUun+K6eIpoFnHvoe\ncstfylSKTCuOqrXQswcVRnzQipMUjgcyyPSsmri697ICO7JKccJDFMTMWfyznZjG\nxsTtiCnWQGNvlBpuE1k9Aq6hSuqDqjdnFbKdSXBOkvN37Av3o8BYwd5k1sP5gZxW\nQg7sagPCTtotpxAuvC4RSo8SxQKBgQDuqw3/zsSk/dnObi4863yr2XtGxl7/U257\nITO55NMtkT8sSy2uj/QCbiOuVZDtAcRk+bVchGFTuzvkBxQhVh2PD7XNcX5R9jAp\n9WTwZUaBD1kZJhAY5HhA9VGllo67CgEzccK+izLtF5Z8BO8cQNNZ41HynWSB0TXY\nqv4I2axnVQKBgQDP0sPrOxEcCPFqnc0qO5dH9Q56sjgQneuw2MDNUzFong1m/xOW\nhVWhVgNeHAliCQG7gvi9Td6J4V+4bBbL/uQZjWCwmOnkea0Jd836BrHBZam7xp02\nlfrqLbHjIQ1OrWAD/SW/G5gYl+jNgZoStfl11I//nnTqechyweUMqu9jCQKBgHNd\nzfuMODPE3UbSrZkW/4GWM+fHNd5Xtvzf1MAT4xyQrCe+A0vxG7MdCDRZ+h8Gjqd8\nqPUarG0sAUw9O7bNiKZBotC2/qehyDBQBUXKvqTeBWSJqJhPv+0uaBQRzdaNhH0Y\njVlLsgc3Mq+qNubCUyeNIWyUNiQmO9AK/5bJmXOtAoGAHrdF8CzwmxLKcdjqIKIt\n3zhU6WjEAR5tlFsfPpujZwBz9nFnGpG0AaNQMdyPQx+vdIa+UEA0gdkPi3gacKJd\nlAEeyp2PmyOxdsW3+pWo5htP+pXgffSErC8ZLRzQsYLYnHIVk01jePx3yCG6sfWf\n85lTkR05DfGAVHjwlJ/s83I=\n-----END PRIVATE KEY-----\n"}]': finished 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.013628+0000 mon.b (mon.2) 188 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.014082+0000 mgr.x (mgr.34643) 34 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.014341+0000 mon.b (mon.2) 189 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.014950+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]: dispatch 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: audit 2024-02-10T09:32:02.018464+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"False","who":"mgr","name":"mgr/dashboard/GRAFANA_API_SSL_VERIFY"}]': finished 2024-02-10T09:32:03.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:02 smithi153 bash[35780]: cephadm 2024-02-10T09:32:02.027405+0000 mgr.x (mgr.34643) 35 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:32:04.026 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:03] "GET /metrics HTTP/1.1" 200 153183 "" "Prometheus/2.33.4" 2024-02-10T09:32:04.027 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:04 smithi204 bash[35809]: cluster 2024-02-10T09:32:02.721864+0000 mgr.x (mgr.34643) 36 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:04 smithi153 bash[39117]: cluster 2024-02-10T09:32:02.721864+0000 mgr.x (mgr.34643) 36 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:04 smithi153 bash[35780]: cluster 2024-02-10T09:32:02.721864+0000 mgr.x (mgr.34643) 36 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:32:04 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:32:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:06 smithi204 bash[35809]: cluster 2024-02-10T09:32:04.722446+0000 mgr.x (mgr.34643) 37 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:06.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:06 smithi153 bash[35780]: cluster 2024-02-10T09:32:04.722446+0000 mgr.x (mgr.34643) 37 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:06.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:06 smithi153 bash[39117]: cluster 2024-02-10T09:32:04.722446+0000 mgr.x (mgr.34643) 37 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:08 smithi204 bash[35809]: cluster 2024-02-10T09:32:06.722788+0000 mgr.x (mgr.34643) 38 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:08 smithi153 bash[35780]: cluster 2024-02-10T09:32:06.722788+0000 mgr.x (mgr.34643) 38 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:08.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:08 smithi153 bash[39117]: cluster 2024-02-10T09:32:06.722788+0000 mgr.x (mgr.34643) 38 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:09.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:32:09 smithi153 bash[59946]: time="2024-02-10T09:32:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:10.303 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:10 smithi153 bash[35780]: cluster 2024-02-10T09:32:08.723355+0000 mgr.x (mgr.34643) 39 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:10.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:10 smithi153 bash[39117]: cluster 2024-02-10T09:32:08.723355+0000 mgr.x (mgr.34643) 39 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:10 smithi204 bash[35809]: cluster 2024-02-10T09:32:08.723355+0000 mgr.x (mgr.34643) 39 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:10.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:32:09 smithi204 bash[48159]: time="2024-02-10T09:32:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:10.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:32:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:32:10.303Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003877386s 2024-02-10T09:32:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:12 smithi204 bash[35809]: cluster 2024-02-10T09:32:10.723729+0000 mgr.x (mgr.34643) 40 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:12 smithi153 bash[35780]: cluster 2024-02-10T09:32:10.723729+0000 mgr.x (mgr.34643) 40 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:12.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:12 smithi153 bash[39117]: cluster 2024-02-10T09:32:10.723729+0000 mgr.x (mgr.34643) 40 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:13.738 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:13] "GET /metrics HTTP/1.1" 200 153191 "" "Prometheus/2.33.4" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 systemd[1]: Stopping Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped Stream Manager" 2024-02-10T09:32:14.057 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2024-02-10T09:32:14.058 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:13 smithi204 bash[45306]: t=2024-02-10T09:32:13+0000 lvl=eror msg="Server shutdown" logger=server reason=nil 2024-02-10T09:32:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:14 smithi204 bash[35809]: cluster 2024-02-10T09:32:12.724102+0000 mgr.x (mgr.34643) 41 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:14.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:14 smithi153 bash[39117]: cluster 2024-02-10T09:32:12.724102+0000 mgr.x (mgr.34643) 41 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:14.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:14 smithi153 bash[35780]: cluster 2024-02-10T09:32:12.724102+0000 mgr.x (mgr.34643) 41 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:32:14 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:32:14.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 bash[49526]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a 2024-02-10T09:32:14.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 bash[49565]: bash: /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/grafana.a/unit.stop: No such file or directory 2024-02-10T09:32:14.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-02-10T09:32:14.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 45322 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49547 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 45322 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49547 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:32:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:32:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[35809]: cluster 2024-02-10T09:32:14.724949+0000 mgr.x (mgr.34643) 42 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:16.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:16 smithi153 bash[35780]: cluster 2024-02-10T09:32:14.724949+0000 mgr.x (mgr.34643) 42 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:16.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:16 smithi153 bash[39117]: cluster 2024-02-10T09:32:14.724949+0000 mgr.x (mgr.34643) 42 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Starting Grafana" logger=server version=6.6.2 commit=3fa63cfc34 branch=HEAD compiled=2020-02-20T12:03:49+0000 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T09:32:17.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing SqlStore" logger=server 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Starting DB migration" logger=migrator 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-10T09:32:17.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-10T09:32:17.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-10T09:32:17.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-10T09:32:17.086 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-10T09:32:17.087 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T09:32:17.087 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T09:32:17.087 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T09:32:17.087 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-10T09:32:17.088 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T09:32:17.089 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T09:32:17.090 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T09:32:17.091 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-10T09:32:17.092 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-10T09:32:17.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T09:32:17.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-10T09:32:17.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T09:32:17.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T09:32:17.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-10T09:32:17.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T09:32:17.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-10T09:32:17.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing HTTPServer" logger=server 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing PluginManager" logger=server 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Starting plugin search" logger=plugins 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing HooksService" logger=server 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2024-02-10T09:32:17.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing RemoteCache" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing QuotaService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing ServerLockService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing RenderingService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing AlertEngine" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing LoginService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing SearchService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing TracingService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing CleanUpService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing NotificationService" logger=server 2024-02-10T09:32:17.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2024-02-10T09:32:17.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-10T09:32:17.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2024-02-10T09:32:17.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2024-02-10T09:32:17.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin." logger=rendering renderer=phantomJS 2024-02-10T09:32:17.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="Initializing Stream Manager" 2024-02-10T09:32:17.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:32:16 smithi204 bash[49570]: t=2024-02-10T09:32:16+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T09:32:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:18 smithi204 bash[35809]: cluster 2024-02-10T09:32:16.725344+0000 mgr.x (mgr.34643) 43 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:18 smithi153 bash[35780]: cluster 2024-02-10T09:32:16.725344+0000 mgr.x (mgr.34643) 43 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:18.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:18 smithi153 bash[39117]: cluster 2024-02-10T09:32:16.725344+0000 mgr.x (mgr.34643) 43 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:20 smithi204 bash[35809]: cluster 2024-02-10T09:32:18.726082+0000 mgr.x (mgr.34643) 44 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:20.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:20 smithi153 bash[35780]: cluster 2024-02-10T09:32:18.726082+0000 mgr.x (mgr.34643) 44 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:20.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:20 smithi153 bash[39117]: cluster 2024-02-10T09:32:18.726082+0000 mgr.x (mgr.34643) 44 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: cluster 2024-02-10T09:32:20.726711+0000 mgr.x (mgr.34643) 45 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.884466+0000 mon.b (mon.2) 190 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.884691+0000 mon.b (mon.2) 191 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.884880+0000 mon.b (mon.2) 192 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.884934+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.885039+0000 mon.b (mon.2) 193 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.885343+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.885614+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[35780]: audit 2024-02-10T09:32:21.885815+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: cluster 2024-02-10T09:32:20.726711+0000 mgr.x (mgr.34643) 45 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.884466+0000 mon.b (mon.2) 190 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.884691+0000 mon.b (mon.2) 191 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.884880+0000 mon.b (mon.2) 192 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.884934+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.885039+0000 mon.b (mon.2) 193 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.885343+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.885614+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:32:22.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:22 smithi153 bash[39117]: audit 2024-02-10T09:32:21.885815+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: cluster 2024-02-10T09:32:20.726711+0000 mgr.x (mgr.34643) 45 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.884466+0000 mon.b (mon.2) 190 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.884691+0000 mon.b (mon.2) 191 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.884880+0000 mon.b (mon.2) 192 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.884934+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.885039+0000 mon.b (mon.2) 193 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.885343+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.885614+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:32:22.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:22 smithi204 bash[35809]: audit 2024-02-10T09:32:21.885815+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:32:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:23] "GET /metrics HTTP/1.1" 200 153188 "" "Prometheus/2.33.4" 2024-02-10T09:32:24.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:24 smithi153 bash[35780]: cluster 2024-02-10T09:32:22.727311+0000 mgr.x (mgr.34643) 46 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:24.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:24 smithi153 bash[39117]: cluster 2024-02-10T09:32:22.727311+0000 mgr.x (mgr.34643) 46 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:24.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:24 smithi204 bash[35809]: cluster 2024-02-10T09:32:22.727311+0000 mgr.x (mgr.34643) 46 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:32:24 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:32:26.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:26 smithi153 bash[35780]: cluster 2024-02-10T09:32:24.728181+0000 mgr.x (mgr.34643) 47 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:26.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:26 smithi153 bash[39117]: cluster 2024-02-10T09:32:24.728181+0000 mgr.x (mgr.34643) 47 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:26.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:26 smithi204 bash[35809]: cluster 2024-02-10T09:32:24.728181+0000 mgr.x (mgr.34643) 47 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:28.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:28 smithi153 bash[35780]: cluster 2024-02-10T09:32:26.728754+0000 mgr.x (mgr.34643) 48 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:28.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:28 smithi153 bash[39117]: cluster 2024-02-10T09:32:26.728754+0000 mgr.x (mgr.34643) 48 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:28.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:28 smithi204 bash[35809]: cluster 2024-02-10T09:32:26.728754+0000 mgr.x (mgr.34643) 48 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:30.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:30 smithi153 bash[35780]: cluster 2024-02-10T09:32:28.729510+0000 mgr.x (mgr.34643) 49 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:30.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:30 smithi153 bash[39117]: cluster 2024-02-10T09:32:28.729510+0000 mgr.x (mgr.34643) 49 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:30.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:30 smithi204 bash[35809]: cluster 2024-02-10T09:32:28.729510+0000 mgr.x (mgr.34643) 49 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:32.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:32 smithi153 bash[35780]: cluster 2024-02-10T09:32:30.729962+0000 mgr.x (mgr.34643) 50 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:32.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:32 smithi153 bash[39117]: cluster 2024-02-10T09:32:30.729962+0000 mgr.x (mgr.34643) 50 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:32.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:32 smithi204 bash[35809]: cluster 2024-02-10T09:32:30.729962+0000 mgr.x (mgr.34643) 50 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:33] "GET /metrics HTTP/1.1" 200 153188 "" "Prometheus/2.33.4" 2024-02-10T09:32:34.411 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:34 smithi153 bash[35780]: cluster 2024-02-10T09:32:32.730398+0000 mgr.x (mgr.34643) 51 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:34.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:34 smithi153 bash[39117]: cluster 2024-02-10T09:32:32.730398+0000 mgr.x (mgr.34643) 51 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:34.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:34 smithi204 bash[35809]: cluster 2024-02-10T09:32:32.730398+0000 mgr.x (mgr.34643) 51 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:34.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:32:34 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:32:36.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:36 smithi153 bash[35780]: cluster 2024-02-10T09:32:34.730988+0000 mgr.x (mgr.34643) 52 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:36.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:36 smithi153 bash[39117]: cluster 2024-02-10T09:32:34.730988+0000 mgr.x (mgr.34643) 52 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:36.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:36 smithi204 bash[35809]: cluster 2024-02-10T09:32:34.730988+0000 mgr.x (mgr.34643) 52 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:38.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:38 smithi153 bash[35780]: cluster 2024-02-10T09:32:36.731538+0000 mgr.x (mgr.34643) 53 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:38.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:38 smithi153 bash[39117]: cluster 2024-02-10T09:32:36.731538+0000 mgr.x (mgr.34643) 53 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:38.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:38 smithi204 bash[35809]: cluster 2024-02-10T09:32:36.731538+0000 mgr.x (mgr.34643) 53 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:40.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:40 smithi153 bash[35780]: cluster 2024-02-10T09:32:38.732251+0000 mgr.x (mgr.34643) 54 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:40.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:40 smithi153 bash[39117]: cluster 2024-02-10T09:32:38.732251+0000 mgr.x (mgr.34643) 54 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:40.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:40 smithi204 bash[35809]: cluster 2024-02-10T09:32:38.732251+0000 mgr.x (mgr.34643) 54 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:42.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:42 smithi153 bash[35780]: cluster 2024-02-10T09:32:40.732672+0000 mgr.x (mgr.34643) 55 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:42.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:42 smithi153 bash[39117]: cluster 2024-02-10T09:32:40.732672+0000 mgr.x (mgr.34643) 55 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:42.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:42 smithi204 bash[35809]: cluster 2024-02-10T09:32:40.732672+0000 mgr.x (mgr.34643) 55 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:43] "GET /metrics HTTP/1.1" 200 153182 "" "Prometheus/2.33.4" 2024-02-10T09:32:44.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:44 smithi153 bash[39117]: cluster 2024-02-10T09:32:42.733123+0000 mgr.x (mgr.34643) 56 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:44.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:44 smithi153 bash[35780]: cluster 2024-02-10T09:32:42.733123+0000 mgr.x (mgr.34643) 56 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:44.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:44 smithi204 bash[35809]: cluster 2024-02-10T09:32:42.733123+0000 mgr.x (mgr.34643) 56 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:44.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:32:44 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:32:46.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:46 smithi153 bash[35780]: cluster 2024-02-10T09:32:44.733826+0000 mgr.x (mgr.34643) 57 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:46.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:46 smithi153 bash[39117]: cluster 2024-02-10T09:32:44.733826+0000 mgr.x (mgr.34643) 57 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:46.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:46 smithi204 bash[35809]: cluster 2024-02-10T09:32:44.733826+0000 mgr.x (mgr.34643) 57 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:47.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:32:46 smithi153 bash[59946]: time="2024-02-10T09:32:46Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:47.749 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:32:47 smithi204 bash[48159]: time="2024-02-10T09:32:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:48.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:47 smithi204 bash[35809]: cluster 2024-02-10T09:32:46.734392+0000 mgr.x (mgr.34643) 58 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:47 smithi153 bash[35780]: cluster 2024-02-10T09:32:46.734392+0000 mgr.x (mgr.34643) 58 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:47 smithi153 bash[39117]: cluster 2024-02-10T09:32:46.734392+0000 mgr.x (mgr.34643) 58 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:49 smithi204 bash[35809]: cluster 2024-02-10T09:32:48.735189+0000 mgr.x (mgr.34643) 59 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:49 smithi153 bash[35780]: cluster 2024-02-10T09:32:48.735189+0000 mgr.x (mgr.34643) 59 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:49 smithi153 bash[39117]: cluster 2024-02-10T09:32:48.735189+0000 mgr.x (mgr.34643) 59 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:51 smithi204 bash[35809]: cluster 2024-02-10T09:32:50.735637+0000 mgr.x (mgr.34643) 60 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:51 smithi153 bash[35780]: cluster 2024-02-10T09:32:50.735637+0000 mgr.x (mgr.34643) 60 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:51 smithi153 bash[39117]: cluster 2024-02-10T09:32:50.735637+0000 mgr.x (mgr.34643) 60 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:53 smithi204 bash[35809]: cluster 2024-02-10T09:32:52.736243+0000 mgr.x (mgr.34643) 61 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:32:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:53] "GET /metrics HTTP/1.1" 200 153182 "" "Prometheus/2.33.4" 2024-02-10T09:32:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:53 smithi153 bash[35780]: cluster 2024-02-10T09:32:52.736243+0000 mgr.x (mgr.34643) 61 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:53 smithi153 bash[39117]: cluster 2024-02-10T09:32:52.736243+0000 mgr.x (mgr.34643) 61 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:32:54 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:32:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:32:56.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:55 smithi204 bash[35809]: cluster 2024-02-10T09:32:54.736960+0000 mgr.x (mgr.34643) 62 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:55 smithi153 bash[35780]: cluster 2024-02-10T09:32:54.736960+0000 mgr.x (mgr.34643) 62 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:55 smithi153 bash[39117]: cluster 2024-02-10T09:32:54.736960+0000 mgr.x (mgr.34643) 62 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:58.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:57 smithi204 bash[35809]: cluster 2024-02-10T09:32:56.737593+0000 mgr.x (mgr.34643) 63 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:57 smithi153 bash[35780]: cluster 2024-02-10T09:32:56.737593+0000 mgr.x (mgr.34643) 63 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:32:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:57 smithi153 bash[39117]: cluster 2024-02-10T09:32:56.737593+0000 mgr.x (mgr.34643) 63 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:32:59 smithi204 bash[35809]: cluster 2024-02-10T09:32:58.738582+0000 mgr.x (mgr.34643) 64 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:32:59 smithi153 bash[35780]: cluster 2024-02-10T09:32:58.738582+0000 mgr.x (mgr.34643) 64 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:32:59 smithi153 bash[39117]: cluster 2024-02-10T09:32:58.738582+0000 mgr.x (mgr.34643) 64 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:01 smithi204 bash[35809]: cluster 2024-02-10T09:33:00.739201+0000 mgr.x (mgr.34643) 65 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:01 smithi153 bash[35780]: cluster 2024-02-10T09:33:00.739201+0000 mgr.x (mgr.34643) 65 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:01 smithi153 bash[39117]: cluster 2024-02-10T09:33:00.739201+0000 mgr.x (mgr.34643) 65 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:04.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:03 smithi204 bash[35809]: cluster 2024-02-10T09:33:02.739667+0000 mgr.x (mgr.34643) 66 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:33:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:03] "GET /metrics HTTP/1.1" 200 153182 "" "Prometheus/2.33.4" 2024-02-10T09:33:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:03 smithi153 bash[35780]: cluster 2024-02-10T09:33:02.739667+0000 mgr.x (mgr.34643) 66 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:03 smithi153 bash[39117]: cluster 2024-02-10T09:33:02.739667+0000 mgr.x (mgr.34643) 66 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:33:04 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:33:06.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:05 smithi204 bash[35809]: cluster 2024-02-10T09:33:04.740538+0000 mgr.x (mgr.34643) 67 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:05 smithi153 bash[35780]: cluster 2024-02-10T09:33:04.740538+0000 mgr.x (mgr.34643) 67 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:05 smithi153 bash[39117]: cluster 2024-02-10T09:33:04.740538+0000 mgr.x (mgr.34643) 67 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:08.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:07 smithi204 bash[35809]: cluster 2024-02-10T09:33:06.741041+0000 mgr.x (mgr.34643) 68 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:07 smithi153 bash[35780]: cluster 2024-02-10T09:33:06.741041+0000 mgr.x (mgr.34643) 68 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:07 smithi153 bash[39117]: cluster 2024-02-10T09:33:06.741041+0000 mgr.x (mgr.34643) 68 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:10.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:09 smithi204 bash[35809]: cluster 2024-02-10T09:33:08.741774+0000 mgr.x (mgr.34643) 69 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:09 smithi153 bash[35780]: cluster 2024-02-10T09:33:08.741774+0000 mgr.x (mgr.34643) 69 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:09 smithi153 bash[39117]: cluster 2024-02-10T09:33:08.741774+0000 mgr.x (mgr.34643) 69 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:12.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:11 smithi204 bash[35809]: cluster 2024-02-10T09:33:10.742250+0000 mgr.x (mgr.34643) 70 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:11 smithi153 bash[35780]: cluster 2024-02-10T09:33:10.742250+0000 mgr.x (mgr.34643) 70 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:11 smithi153 bash[39117]: cluster 2024-02-10T09:33:10.742250+0000 mgr.x (mgr.34643) 70 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:13 smithi204 bash[35809]: cluster 2024-02-10T09:33:12.742696+0000 mgr.x (mgr.34643) 71 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:14.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:33:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:13] "GET /metrics HTTP/1.1" 200 153178 "" "Prometheus/2.33.4" 2024-02-10T09:33:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:13 smithi153 bash[35780]: cluster 2024-02-10T09:33:12.742696+0000 mgr.x (mgr.34643) 71 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:13 smithi153 bash[39117]: cluster 2024-02-10T09:33:12.742696+0000 mgr.x (mgr.34643) 71 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:33:14 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:33:16.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:15 smithi204 bash[35809]: cluster 2024-02-10T09:33:14.743390+0000 mgr.x (mgr.34643) 72 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:15 smithi153 bash[35780]: cluster 2024-02-10T09:33:14.743390+0000 mgr.x (mgr.34643) 72 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:15 smithi153 bash[39117]: cluster 2024-02-10T09:33:14.743390+0000 mgr.x (mgr.34643) 72 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:17 smithi204 bash[35809]: cluster 2024-02-10T09:33:16.743771+0000 mgr.x (mgr.34643) 73 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:17 smithi153 bash[39117]: cluster 2024-02-10T09:33:16.743771+0000 mgr.x (mgr.34643) 73 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:17 smithi153 bash[35780]: cluster 2024-02-10T09:33:16.743771+0000 mgr.x (mgr.34643) 73 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:20.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:19 smithi204 bash[35809]: cluster 2024-02-10T09:33:18.744583+0000 mgr.x (mgr.34643) 74 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:19 smithi153 bash[35780]: cluster 2024-02-10T09:33:18.744583+0000 mgr.x (mgr.34643) 74 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:19 smithi153 bash[39117]: cluster 2024-02-10T09:33:18.744583+0000 mgr.x (mgr.34643) 74 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:20.941 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T09:33:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:21 smithi153 bash[35780]: cluster 2024-02-10T09:33:20.745221+0000 mgr.x (mgr.34643) 75 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:21.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:21 smithi153 bash[39117]: cluster 2024-02-10T09:33:20.745221+0000 mgr.x (mgr.34643) 75 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:21 smithi204 bash[35809]: cluster 2024-02-10T09:33:20.745221+0000 mgr.x (mgr.34643) 75 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.888396+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.888618+0000 mon.b (mon.2) 195 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.888825+0000 mon.b (mon.2) 196 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.888905+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.889028+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.889231+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.889376+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-02-10T09:33:22.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[39117]: audit 2024-02-10T09:33:21.889560+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.888396+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.888618+0000 mon.b (mon.2) 195 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.888825+0000 mon.b (mon.2) 196 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.888905+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.889028+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.889231+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.889376+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-02-10T09:33:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:22 smithi153 bash[35780]: audit 2024-02-10T09:33:21.889560+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:33:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.888396+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:33:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.888618+0000 mon.b (mon.2) 195 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:33:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.888825+0000 mon.b (mon.2) 196 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-02-10T09:33:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.888905+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:33:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.889028+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:33:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.889231+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:33:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.889376+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-02-10T09:33:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:22 smithi204 bash[35809]: audit 2024-02-10T09:33:21.889560+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:33:23.448 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled to redeploy mgr.y on host 'smithi153' 2024-02-10T09:33:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:33:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:23] "GET /metrics HTTP/1.1" 200 153192 "" "Prometheus/2.33.4" 2024-02-10T09:33:24.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:23 smithi204 bash[35809]: cluster 2024-02-10T09:33:22.745826+0000 mgr.x (mgr.34643) 76 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:24.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:23 smithi204 bash[35809]: audit 2024-02-10T09:33:23.127052+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.153:0/2854112038' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:33:24.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:23 smithi204 bash[35809]: audit 2024-02-10T09:33:23.441880+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]: dispatch 2024-02-10T09:33:24.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:23 smithi204 bash[35809]: audit 2024-02-10T09:33:23.442432+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]: dispatch 2024-02-10T09:33:24.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:23 smithi204 bash[35809]: audit 2024-02-10T09:33:23.446413+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]': finished 2024-02-10T09:33:24.112 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[35780]: cluster 2024-02-10T09:33:22.745826+0000 mgr.x (mgr.34643) 76 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[35780]: audit 2024-02-10T09:33:23.127052+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.153:0/2854112038' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[35780]: audit 2024-02-10T09:33:23.441880+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]: dispatch 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[35780]: audit 2024-02-10T09:33:23.442432+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]: dispatch 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[35780]: audit 2024-02-10T09:33:23.446413+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]': finished 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[39117]: cluster 2024-02-10T09:33:22.745826+0000 mgr.x (mgr.34643) 76 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[39117]: audit 2024-02-10T09:33:23.127052+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.153:0/2854112038' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[39117]: audit 2024-02-10T09:33:23.441880+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]: dispatch 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[39117]: audit 2024-02-10T09:33:23.442432+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]: dispatch 2024-02-10T09:33:24.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:23 smithi153 bash[39117]: audit 2024-02-10T09:33:23.446413+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "who": "mgr.y"}]': finished 2024-02-10T09:33:24.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps --refresh' 2024-02-10T09:33:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:33:24 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:33:24.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:33:24 smithi153 bash[59946]: time="2024-02-10T09:33:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:33:25.035 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:24 smithi153 bash[35780]: audit 2024-02-10T09:33:23.440912+0000 mgr.x (mgr.34643) 77 : audit [DBG] from='client.24760 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:33:25.036 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:24 smithi153 bash[35780]: cephadm 2024-02-10T09:33:23.447559+0000 mgr.x (mgr.34643) 78 : cephadm [INF] Schedule redeploy daemon mgr.y 2024-02-10T09:33:25.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:24 smithi153 bash[39117]: audit 2024-02-10T09:33:23.440912+0000 mgr.x (mgr.34643) 77 : audit [DBG] from='client.24760 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:33:25.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:24 smithi153 bash[39117]: cephadm 2024-02-10T09:33:23.447559+0000 mgr.x (mgr.34643) 78 : cephadm [INF] Schedule redeploy daemon mgr.y 2024-02-10T09:33:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:24 smithi204 bash[35809]: audit 2024-02-10T09:33:23.440912+0000 mgr.x (mgr.34643) 77 : audit [DBG] from='client.24760 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:33:25.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:24 smithi204 bash[35809]: cephadm 2024-02-10T09:33:23.447559+0000 mgr.x (mgr.34643) 78 : cephadm [INF] Schedule redeploy daemon mgr.y 2024-02-10T09:33:25.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:33:24 smithi204 bash[48159]: time="2024-02-10T09:33:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 starting - - - - 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (4m) 93s ago 4m 29.7M - 6.6.2 87a51ecf0b1c a148afe5b831 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (6m) 93s ago 14m 417M - 16.2.14-521-g0e714d9a 1fbc92c859dc 46c6073a79be 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 running (16m) 2m ago 16m 287M - 15.2.0 204a01f9b0b6 03548f93e5ac 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (16m) 2m ago 16m 92.7M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (15m) 93s ago 15m 101M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (15m) 2m ago 15m 102M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 error 2m ago 7m - - 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running 93s ago 7m - - 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (14m) 2m ago 14m 42.1M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:33:25.994 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (13m) 2m ago 14m 42.0M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (13m) 2m ago 13m 40.4M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (13m) 2m ago 13m 43.9M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (13m) 93s ago 13m 45.8M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (12m) 93s ago 12m 46.0M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (12m) 93s ago 12m 43.7M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (12m) 93s ago 12m 44.5M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (98s) 93s ago 97s 31.6M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (4m) 2m ago 4m 31.8M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:33:25.995 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (4m) 93s ago 4m 33.1M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:33:26.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:25 smithi204 bash[35809]: cluster 2024-02-10T09:33:24.746359+0000 mgr.x (mgr.34643) 79 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:25 smithi153 bash[39117]: cluster 2024-02-10T09:33:24.746359+0000 mgr.x (mgr.34643) 79 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:25 smithi153 bash[35780]: cluster 2024-02-10T09:33:24.746359+0000 mgr.x (mgr.34643) 79 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:26.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 180' 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.980805+0000 mgr.x (mgr.34643) 80 : audit [DBG] from='client.24766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.983059+0000 mon.b (mon.2) 199 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.983467+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.983838+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.984177+0000 mon.b (mon.2) 202 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.984526+0000 mon.b (mon.2) 203 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.984898+0000 mon.b (mon.2) 204 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.986831+0000 mon.b (mon.2) 205 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:26.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.987213+0000 mon.b (mon.2) 206 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.987585+0000 mon.b (mon.2) 207 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.987917+0000 mon.b (mon.2) 208 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[39117]: audit 2024-02-10T09:33:25.988250+0000 mon.b (mon.2) 209 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.980805+0000 mgr.x (mgr.34643) 80 : audit [DBG] from='client.24766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.983059+0000 mon.b (mon.2) 199 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.983467+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.983838+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.984177+0000 mon.b (mon.2) 202 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.984526+0000 mon.b (mon.2) 203 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.984898+0000 mon.b (mon.2) 204 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.986831+0000 mon.b (mon.2) 205 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:26.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.987213+0000 mon.b (mon.2) 206 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.987585+0000 mon.b (mon.2) 207 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.987917+0000 mon.b (mon.2) 208 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:26.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:26 smithi153 bash[35780]: audit 2024-02-10T09:33:25.988250+0000 mon.b (mon.2) 209 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.980805+0000 mgr.x (mgr.34643) 80 : audit [DBG] from='client.24766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:33:27.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.983059+0000 mon.b (mon.2) 199 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:27.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.983467+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.983838+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.984177+0000 mon.b (mon.2) 202 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.984526+0000 mon.b (mon.2) 203 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.984898+0000 mon.b (mon.2) 204 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.986831+0000 mon.b (mon.2) 205 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.987213+0000 mon.b (mon.2) 206 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.987585+0000 mon.b (mon.2) 207 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.987917+0000 mon.b (mon.2) 208 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:27.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:26 smithi204 bash[35809]: audit 2024-02-10T09:33:25.988250+0000 mon.b (mon.2) 209 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:33:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:27 smithi204 bash[35809]: cluster 2024-02-10T09:33:26.746783+0000 mgr.x (mgr.34643) 81 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:27 smithi153 bash[35780]: cluster 2024-02-10T09:33:26.746783+0000 mgr.x (mgr.34643) 81 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:27 smithi153 bash[39117]: cluster 2024-02-10T09:33:26.746783+0000 mgr.x (mgr.34643) 81 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:30.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:29 smithi204 bash[35809]: cluster 2024-02-10T09:33:28.747530+0000 mgr.x (mgr.34643) 82 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:29 smithi153 bash[35780]: cluster 2024-02-10T09:33:28.747530+0000 mgr.x (mgr.34643) 82 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:29 smithi153 bash[39117]: cluster 2024-02-10T09:33:28.747530+0000 mgr.x (mgr.34643) 82 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:31 smithi204 bash[35809]: cluster 2024-02-10T09:33:30.747992+0000 mgr.x (mgr.34643) 83 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:31 smithi153 bash[39117]: cluster 2024-02-10T09:33:30.747992+0000 mgr.x (mgr.34643) 83 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:31 smithi153 bash[35780]: cluster 2024-02-10T09:33:30.747992+0000 mgr.x (mgr.34643) 83 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:33 smithi204 bash[35809]: cluster 2024-02-10T09:33:32.748797+0000 mgr.x (mgr.34643) 84 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:34.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:33:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:33] "GET /metrics HTTP/1.1" 200 153192 "" "Prometheus/2.33.4" 2024-02-10T09:33:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:33 smithi153 bash[35780]: cluster 2024-02-10T09:33:32.748797+0000 mgr.x (mgr.34643) 84 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:33 smithi153 bash[39117]: cluster 2024-02-10T09:33:32.748797+0000 mgr.x (mgr.34643) 84 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:33:34 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:33:36.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:35 smithi204 bash[35809]: cluster 2024-02-10T09:33:34.749421+0000 mgr.x (mgr.34643) 85 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:35 smithi153 bash[35780]: cluster 2024-02-10T09:33:34.749421+0000 mgr.x (mgr.34643) 85 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:35 smithi153 bash[39117]: cluster 2024-02-10T09:33:34.749421+0000 mgr.x (mgr.34643) 85 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:37 smithi204 bash[35809]: cluster 2024-02-10T09:33:36.749782+0000 mgr.x (mgr.34643) 86 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:37 smithi153 bash[35780]: cluster 2024-02-10T09:33:36.749782+0000 mgr.x (mgr.34643) 86 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:37 smithi153 bash[39117]: cluster 2024-02-10T09:33:36.749782+0000 mgr.x (mgr.34643) 86 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:39 smithi204 bash[35809]: cluster 2024-02-10T09:33:38.750405+0000 mgr.x (mgr.34643) 87 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:39 smithi153 bash[35780]: cluster 2024-02-10T09:33:38.750405+0000 mgr.x (mgr.34643) 87 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:39 smithi153 bash[39117]: cluster 2024-02-10T09:33:38.750405+0000 mgr.x (mgr.34643) 87 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:41 smithi204 bash[35809]: cluster 2024-02-10T09:33:40.750917+0000 mgr.x (mgr.34643) 88 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:41 smithi153 bash[39117]: cluster 2024-02-10T09:33:40.750917+0000 mgr.x (mgr.34643) 88 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:41 smithi153 bash[35780]: cluster 2024-02-10T09:33:40.750917+0000 mgr.x (mgr.34643) 88 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:43 smithi204 bash[35809]: cluster 2024-02-10T09:33:42.751552+0000 mgr.x (mgr.34643) 89 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:44.154 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:33:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:43] "GET /metrics HTTP/1.1" 200 153203 "" "Prometheus/2.33.4" 2024-02-10T09:33:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:43 smithi153 bash[35780]: cluster 2024-02-10T09:33:42.751552+0000 mgr.x (mgr.34643) 89 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:43 smithi153 bash[39117]: cluster 2024-02-10T09:33:42.751552+0000 mgr.x (mgr.34643) 89 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:33:44 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:33:46.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:45 smithi204 bash[35809]: cluster 2024-02-10T09:33:44.752084+0000 mgr.x (mgr.34643) 90 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:45 smithi153 bash[35780]: cluster 2024-02-10T09:33:44.752084+0000 mgr.x (mgr.34643) 90 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:45 smithi153 bash[39117]: cluster 2024-02-10T09:33:44.752084+0000 mgr.x (mgr.34643) 90 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:48.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:47 smithi204 bash[35809]: cluster 2024-02-10T09:33:46.752560+0000 mgr.x (mgr.34643) 91 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:47 smithi153 bash[39117]: cluster 2024-02-10T09:33:46.752560+0000 mgr.x (mgr.34643) 91 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:47 smithi153 bash[35780]: cluster 2024-02-10T09:33:46.752560+0000 mgr.x (mgr.34643) 91 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:49 smithi204 bash[35809]: cluster 2024-02-10T09:33:48.753297+0000 mgr.x (mgr.34643) 92 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:49 smithi153 bash[39117]: cluster 2024-02-10T09:33:48.753297+0000 mgr.x (mgr.34643) 92 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:49 smithi153 bash[35780]: cluster 2024-02-10T09:33:48.753297+0000 mgr.x (mgr.34643) 92 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:51 smithi204 bash[35809]: cluster 2024-02-10T09:33:50.753740+0000 mgr.x (mgr.34643) 93 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:51 smithi153 bash[35780]: cluster 2024-02-10T09:33:50.753740+0000 mgr.x (mgr.34643) 93 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:51 smithi153 bash[39117]: cluster 2024-02-10T09:33:50.753740+0000 mgr.x (mgr.34643) 93 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:33:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:53] "GET /metrics HTTP/1.1" 200 153204 "" "Prometheus/2.33.4" 2024-02-10T09:33:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:53 smithi204 bash[35809]: cluster 2024-02-10T09:33:52.754450+0000 mgr.x (mgr.34643) 94 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:53 smithi153 bash[35780]: cluster 2024-02-10T09:33:52.754450+0000 mgr.x (mgr.34643) 94 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:53 smithi153 bash[39117]: cluster 2024-02-10T09:33:52.754450+0000 mgr.x (mgr.34643) 94 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:33:54 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:33:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:33:56.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:55 smithi204 bash[35809]: cluster 2024-02-10T09:33:54.754993+0000 mgr.x (mgr.34643) 95 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:55 smithi153 bash[35780]: cluster 2024-02-10T09:33:54.754993+0000 mgr.x (mgr.34643) 95 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:55 smithi153 bash[39117]: cluster 2024-02-10T09:33:54.754993+0000 mgr.x (mgr.34643) 95 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:58.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:57 smithi204 bash[35809]: cluster 2024-02-10T09:33:56.755580+0000 mgr.x (mgr.34643) 96 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:57 smithi153 bash[35780]: cluster 2024-02-10T09:33:56.755580+0000 mgr.x (mgr.34643) 96 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:33:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:57 smithi153 bash[39117]: cluster 2024-02-10T09:33:56.755580+0000 mgr.x (mgr.34643) 96 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:33:59 smithi204 bash[35809]: cluster 2024-02-10T09:33:58.756390+0000 mgr.x (mgr.34643) 97 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:00.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:33:59 smithi153 bash[35780]: cluster 2024-02-10T09:33:58.756390+0000 mgr.x (mgr.34643) 97 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:33:59 smithi153 bash[39117]: cluster 2024-02-10T09:33:58.756390+0000 mgr.x (mgr.34643) 97 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:02.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:01 smithi204 bash[35809]: cluster 2024-02-10T09:34:00.756997+0000 mgr.x (mgr.34643) 98 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:02.152 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:01 smithi153 bash[39117]: cluster 2024-02-10T09:34:00.756997+0000 mgr.x (mgr.34643) 98 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:02.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:01 smithi153 bash[35780]: cluster 2024-02-10T09:34:00.756997+0000 mgr.x (mgr.34643) 98 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:02.153 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 bash[59946]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:34:02.153 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 bash[59946]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:34:02.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 bash[59946]: Error: 3 errors occurred while pulling: 2024-02-10T09:34:02.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 bash[59946]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:34:02.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 bash[59946]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:34:02.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 bash[59946]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:34:02.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Control process exited, code=exited status=125 2024-02-10T09:34:02.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:34:02.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:02 smithi153 systemd[1]: Failed to start Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:34:02.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 bash[48159]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:34:02.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 bash[48159]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:34:03.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 bash[48159]: Error: 3 errors occurred while pulling: 2024-02-10T09:34:03.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 bash[48159]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:34:03.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 bash[48159]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:34:03.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 bash[48159]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:34:03.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=125/n/a 2024-02-10T09:34:03.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:34:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:03 smithi204 bash[35809]: cluster 2024-02-10T09:34:02.757623+0000 mgr.x (mgr.34643) 99 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:34:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:03] "GET /metrics HTTP/1.1" 200 153204 "" "Prometheus/2.33.4" 2024-02-10T09:34:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:03 smithi153 bash[35780]: cluster 2024-02-10T09:34:02.757623+0000 mgr.x (mgr.34643) 99 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:03 smithi153 bash[39117]: cluster 2024-02-10T09:34:02.757623+0000 mgr.x (mgr.34643) 99 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:34:04 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:34:06.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:05 smithi204 bash[35809]: cluster 2024-02-10T09:34:04.758222+0000 mgr.x (mgr.34643) 100 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:05 smithi153 bash[35780]: cluster 2024-02-10T09:34:04.758222+0000 mgr.x (mgr.34643) 100 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:05 smithi153 bash[39117]: cluster 2024-02-10T09:34:04.758222+0000 mgr.x (mgr.34643) 100 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:08.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:07 smithi204 bash[35809]: cluster 2024-02-10T09:34:06.758759+0000 mgr.x (mgr.34643) 101 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:07 smithi153 bash[35780]: cluster 2024-02-10T09:34:06.758759+0000 mgr.x (mgr.34643) 101 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:07 smithi153 bash[39117]: cluster 2024-02-10T09:34:06.758759+0000 mgr.x (mgr.34643) 101 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:10.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:09 smithi204 bash[35809]: cluster 2024-02-10T09:34:08.759450+0000 mgr.x (mgr.34643) 102 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:09 smithi153 bash[35780]: cluster 2024-02-10T09:34:08.759450+0000 mgr.x (mgr.34643) 102 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:09 smithi153 bash[39117]: cluster 2024-02-10T09:34:08.759450+0000 mgr.x (mgr.34643) 102 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:12.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:11 smithi204 bash[35809]: cluster 2024-02-10T09:34:10.759873+0000 mgr.x (mgr.34643) 103 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:11 smithi153 bash[35780]: cluster 2024-02-10T09:34:10.759873+0000 mgr.x (mgr.34643) 103 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:11 smithi153 bash[39117]: cluster 2024-02-10T09:34:10.759873+0000 mgr.x (mgr.34643) 103 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:12.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:12 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:34:12.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:12 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:34:12.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:12 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:34:12.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:12 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:34:12.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:12 smithi153 bash[61050]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:34:12.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:12 smithi153 bash[61050]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:34:13.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:34:13.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:34:13.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:12 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:34:13.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:12 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:34:13.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:12 smithi204 bash[49655]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:34:13.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:12 smithi204 bash[49655]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:34:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:13 smithi204 bash[35809]: cluster 2024-02-10T09:34:12.760594+0000 mgr.x (mgr.34643) 104 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:34:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:13] "GET /metrics HTTP/1.1" 200 153205 "" "Prometheus/2.33.4" 2024-02-10T09:34:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:13 smithi153 bash[35780]: cluster 2024-02-10T09:34:12.760594+0000 mgr.x (mgr.34643) 104 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:13 smithi153 bash[39117]: cluster 2024-02-10T09:34:12.760594+0000 mgr.x (mgr.34643) 104 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:14.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:34:14 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:34:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:15 smithi153 bash[35780]: cluster 2024-02-10T09:34:14.761200+0000 mgr.x (mgr.34643) 105 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:15 smithi153 bash[39117]: cluster 2024-02-10T09:34:14.761200+0000 mgr.x (mgr.34643) 105 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:15 smithi204 bash[35809]: cluster 2024-02-10T09:34:14.761200+0000 mgr.x (mgr.34643) 105 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:17 smithi204 bash[35809]: cluster 2024-02-10T09:34:16.761550+0000 mgr.x (mgr.34643) 106 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:17 smithi153 bash[35780]: cluster 2024-02-10T09:34:16.761550+0000 mgr.x (mgr.34643) 106 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:17 smithi153 bash[39117]: cluster 2024-02-10T09:34:16.761550+0000 mgr.x (mgr.34643) 106 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:20.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:19 smithi204 bash[35809]: cluster 2024-02-10T09:34:18.762425+0000 mgr.x (mgr.34643) 107 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:20.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:19 smithi153 bash[35780]: cluster 2024-02-10T09:34:18.762425+0000 mgr.x (mgr.34643) 107 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:19 smithi153 bash[39117]: cluster 2024-02-10T09:34:18.762425+0000 mgr.x (mgr.34643) 107 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:22.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:21 smithi204 bash[35809]: cluster 2024-02-10T09:34:20.762763+0000 mgr.x (mgr.34643) 108 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:21 smithi153 bash[35780]: cluster 2024-02-10T09:34:20.762763+0000 mgr.x (mgr.34643) 108 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:21 smithi153 bash[39117]: cluster 2024-02-10T09:34:20.762763+0000 mgr.x (mgr.34643) 108 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.892425+0000 mon.b (mon.2) 210 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.892692+0000 mon.b (mon.2) 211 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.892922+0000 mon.b (mon.2) 212 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.893077+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.893157+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.893814+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.894096+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:34:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:22 smithi204 bash[35809]: audit 2024-02-10T09:34:21.894363+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.892425+0000 mon.b (mon.2) 210 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.892692+0000 mon.b (mon.2) 211 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.892922+0000 mon.b (mon.2) 212 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.893077+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.893157+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.893814+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.894096+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[35780]: audit 2024-02-10T09:34:21.894363+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.892425+0000 mon.b (mon.2) 210 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.892692+0000 mon.b (mon.2) 211 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.892922+0000 mon.b (mon.2) 212 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.893077+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.893157+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.893814+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.894096+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:34:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:22 smithi153 bash[39117]: audit 2024-02-10T09:34:21.894363+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-10T09:34:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:23 smithi204 bash[35809]: cluster 2024-02-10T09:34:22.763466+0000 mgr.x (mgr.34643) 109 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:34:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:23] "GET /metrics HTTP/1.1" 200 153205 "" "Prometheus/2.33.4" 2024-02-10T09:34:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:23 smithi153 bash[35780]: cluster 2024-02-10T09:34:22.763466+0000 mgr.x (mgr.34643) 109 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:23 smithi153 bash[39117]: cluster 2024-02-10T09:34:22.763466+0000 mgr.x (mgr.34643) 109 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:34:24 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:34:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:25 smithi153 bash[35780]: cluster 2024-02-10T09:34:24.763828+0000 mgr.x (mgr.34643) 110 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:25 smithi153 bash[39117]: cluster 2024-02-10T09:34:24.763828+0000 mgr.x (mgr.34643) 110 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:25 smithi204 bash[35809]: cluster 2024-02-10T09:34:24.763828+0000 mgr.x (mgr.34643) 110 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:27 smithi204 bash[35809]: cluster 2024-02-10T09:34:26.764375+0000 mgr.x (mgr.34643) 111 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:27 smithi153 bash[35780]: cluster 2024-02-10T09:34:26.764375+0000 mgr.x (mgr.34643) 111 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:27 smithi153 bash[39117]: cluster 2024-02-10T09:34:26.764375+0000 mgr.x (mgr.34643) 111 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:30.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:29 smithi204 bash[35809]: cluster 2024-02-10T09:34:28.765199+0000 mgr.x (mgr.34643) 112 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:29 smithi153 bash[35780]: cluster 2024-02-10T09:34:28.765199+0000 mgr.x (mgr.34643) 112 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:29 smithi153 bash[39117]: cluster 2024-02-10T09:34:28.765199+0000 mgr.x (mgr.34643) 112 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:32.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:31 smithi204 bash[35809]: cluster 2024-02-10T09:34:30.765727+0000 mgr.x (mgr.34643) 113 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:31 smithi153 bash[35780]: cluster 2024-02-10T09:34:30.765727+0000 mgr.x (mgr.34643) 113 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:31 smithi153 bash[39117]: cluster 2024-02-10T09:34:30.765727+0000 mgr.x (mgr.34643) 113 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:33 smithi204 bash[35809]: cluster 2024-02-10T09:34:32.766515+0000 mgr.x (mgr.34643) 114 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:34:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:33] "GET /metrics HTTP/1.1" 200 153205 "" "Prometheus/2.33.4" 2024-02-10T09:34:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:33 smithi153 bash[35780]: cluster 2024-02-10T09:34:32.766515+0000 mgr.x (mgr.34643) 114 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:33 smithi153 bash[39117]: cluster 2024-02-10T09:34:32.766515+0000 mgr.x (mgr.34643) 114 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:34:34 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:34:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:35 smithi153 bash[35780]: cluster 2024-02-10T09:34:34.766976+0000 mgr.x (mgr.34643) 115 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:35 smithi153 bash[39117]: cluster 2024-02-10T09:34:34.766976+0000 mgr.x (mgr.34643) 115 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:35 smithi204 bash[35809]: cluster 2024-02-10T09:34:34.766976+0000 mgr.x (mgr.34643) 115 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:38.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:37 smithi204 bash[35809]: cluster 2024-02-10T09:34:36.767549+0000 mgr.x (mgr.34643) 116 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:37 smithi153 bash[35780]: cluster 2024-02-10T09:34:36.767549+0000 mgr.x (mgr.34643) 116 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:37 smithi153 bash[39117]: cluster 2024-02-10T09:34:36.767549+0000 mgr.x (mgr.34643) 116 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:40.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:39 smithi204 bash[35809]: cluster 2024-02-10T09:34:38.768370+0000 mgr.x (mgr.34643) 117 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:39 smithi153 bash[35780]: cluster 2024-02-10T09:34:38.768370+0000 mgr.x (mgr.34643) 117 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:39 smithi153 bash[39117]: cluster 2024-02-10T09:34:38.768370+0000 mgr.x (mgr.34643) 117 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:42.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:41 smithi204 bash[35809]: cluster 2024-02-10T09:34:40.768862+0000 mgr.x (mgr.34643) 118 : cluster [DBG] pgmap v105: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:41 smithi153 bash[35780]: cluster 2024-02-10T09:34:40.768862+0000 mgr.x (mgr.34643) 118 : cluster [DBG] pgmap v105: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:41 smithi153 bash[39117]: cluster 2024-02-10T09:34:40.768862+0000 mgr.x (mgr.34643) 118 : cluster [DBG] pgmap v105: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:34:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:43] "GET /metrics HTTP/1.1" 200 153204 "" "Prometheus/2.33.4" 2024-02-10T09:34:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:43 smithi204 bash[35809]: cluster 2024-02-10T09:34:42.769794+0000 mgr.x (mgr.34643) 119 : cluster [DBG] pgmap v106: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:43 smithi153 bash[35780]: cluster 2024-02-10T09:34:42.769794+0000 mgr.x (mgr.34643) 119 : cluster [DBG] pgmap v106: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:43 smithi153 bash[39117]: cluster 2024-02-10T09:34:42.769794+0000 mgr.x (mgr.34643) 119 : cluster [DBG] pgmap v106: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:34:44 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:34:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:45 smithi153 bash[35780]: cluster 2024-02-10T09:34:44.770342+0000 mgr.x (mgr.34643) 120 : cluster [DBG] pgmap v107: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:46.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:45 smithi153 bash[39117]: cluster 2024-02-10T09:34:44.770342+0000 mgr.x (mgr.34643) 120 : cluster [DBG] pgmap v107: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:46.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:45 smithi204 bash[35809]: cluster 2024-02-10T09:34:44.770342+0000 mgr.x (mgr.34643) 120 : cluster [DBG] pgmap v107: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:48.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:47 smithi204 bash[35809]: cluster 2024-02-10T09:34:46.770835+0000 mgr.x (mgr.34643) 121 : cluster [DBG] pgmap v108: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:47 smithi153 bash[35780]: cluster 2024-02-10T09:34:46.770835+0000 mgr.x (mgr.34643) 121 : cluster [DBG] pgmap v108: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:47 smithi153 bash[39117]: cluster 2024-02-10T09:34:46.770835+0000 mgr.x (mgr.34643) 121 : cluster [DBG] pgmap v108: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:49.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:34:49 smithi153 bash[61050]: time="2024-02-10T09:34:49Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:34:49.827 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:34:49 smithi204 bash[49655]: time="2024-02-10T09:34:49Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:34:50.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:49 smithi204 bash[35809]: cluster 2024-02-10T09:34:48.771570+0000 mgr.x (mgr.34643) 122 : cluster [DBG] pgmap v109: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:49 smithi153 bash[35780]: cluster 2024-02-10T09:34:48.771570+0000 mgr.x (mgr.34643) 122 : cluster [DBG] pgmap v109: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:49 smithi153 bash[39117]: cluster 2024-02-10T09:34:48.771570+0000 mgr.x (mgr.34643) 122 : cluster [DBG] pgmap v109: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:52.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:51 smithi204 bash[35809]: cluster 2024-02-10T09:34:50.771938+0000 mgr.x (mgr.34643) 123 : cluster [DBG] pgmap v110: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:51 smithi153 bash[35780]: cluster 2024-02-10T09:34:50.771938+0000 mgr.x (mgr.34643) 123 : cluster [DBG] pgmap v110: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:51 smithi153 bash[39117]: cluster 2024-02-10T09:34:50.771938+0000 mgr.x (mgr.34643) 123 : cluster [DBG] pgmap v110: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:53 smithi204 bash[35809]: cluster 2024-02-10T09:34:52.772739+0000 mgr.x (mgr.34643) 124 : cluster [DBG] pgmap v111: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:54.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:34:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:53] "GET /metrics HTTP/1.1" 200 153203 "" "Prometheus/2.33.4" 2024-02-10T09:34:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:53 smithi153 bash[35780]: cluster 2024-02-10T09:34:52.772739+0000 mgr.x (mgr.34643) 124 : cluster [DBG] pgmap v111: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:53 smithi153 bash[39117]: cluster 2024-02-10T09:34:52.772739+0000 mgr.x (mgr.34643) 124 : cluster [DBG] pgmap v111: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:34:54 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:34:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:34:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:55 smithi153 bash[35780]: cluster 2024-02-10T09:34:54.773144+0000 mgr.x (mgr.34643) 125 : cluster [DBG] pgmap v112: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:55 smithi153 bash[39117]: cluster 2024-02-10T09:34:54.773144+0000 mgr.x (mgr.34643) 125 : cluster [DBG] pgmap v112: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:55 smithi204 bash[35809]: cluster 2024-02-10T09:34:54.773144+0000 mgr.x (mgr.34643) 125 : cluster [DBG] pgmap v112: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:58.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:57 smithi204 bash[35809]: cluster 2024-02-10T09:34:56.773497+0000 mgr.x (mgr.34643) 126 : cluster [DBG] pgmap v113: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:57 smithi153 bash[35780]: cluster 2024-02-10T09:34:56.773497+0000 mgr.x (mgr.34643) 126 : cluster [DBG] pgmap v113: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:34:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:57 smithi153 bash[39117]: cluster 2024-02-10T09:34:56.773497+0000 mgr.x (mgr.34643) 126 : cluster [DBG] pgmap v113: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:00.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:34:59 smithi204 bash[35809]: cluster 2024-02-10T09:34:58.774273+0000 mgr.x (mgr.34643) 127 : cluster [DBG] pgmap v114: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:00.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:34:59 smithi153 bash[35780]: cluster 2024-02-10T09:34:58.774273+0000 mgr.x (mgr.34643) 127 : cluster [DBG] pgmap v114: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:00.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:34:59 smithi153 bash[39117]: cluster 2024-02-10T09:34:58.774273+0000 mgr.x (mgr.34643) 127 : cluster [DBG] pgmap v114: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:01 smithi153 bash[35780]: cluster 2024-02-10T09:35:00.774788+0000 mgr.x (mgr.34643) 128 : cluster [DBG] pgmap v115: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:02.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:01 smithi153 bash[39117]: cluster 2024-02-10T09:35:00.774788+0000 mgr.x (mgr.34643) 128 : cluster [DBG] pgmap v115: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:01 smithi204 bash[35809]: cluster 2024-02-10T09:35:00.774788+0000 mgr.x (mgr.34643) 128 : cluster [DBG] pgmap v115: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:04.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:03 smithi204 bash[35809]: cluster 2024-02-10T09:35:02.775657+0000 mgr.x (mgr.34643) 129 : cluster [DBG] pgmap v116: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:03] "GET /metrics HTTP/1.1" 200 153203 "" "Prometheus/2.33.4" 2024-02-10T09:35:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:03 smithi153 bash[35780]: cluster 2024-02-10T09:35:02.775657+0000 mgr.x (mgr.34643) 129 : cluster [DBG] pgmap v116: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:03 smithi153 bash[39117]: cluster 2024-02-10T09:35:02.775657+0000 mgr.x (mgr.34643) 129 : cluster [DBG] pgmap v116: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:04 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:35:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:05 smithi153 bash[35780]: cluster 2024-02-10T09:35:04.776141+0000 mgr.x (mgr.34643) 130 : cluster [DBG] pgmap v117: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:05 smithi153 bash[39117]: cluster 2024-02-10T09:35:04.776141+0000 mgr.x (mgr.34643) 130 : cluster [DBG] pgmap v117: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:05 smithi204 bash[35809]: cluster 2024-02-10T09:35:04.776141+0000 mgr.x (mgr.34643) 130 : cluster [DBG] pgmap v117: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:08.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:07 smithi204 bash[35809]: cluster 2024-02-10T09:35:06.776652+0000 mgr.x (mgr.34643) 131 : cluster [DBG] pgmap v118: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:07 smithi153 bash[35780]: cluster 2024-02-10T09:35:06.776652+0000 mgr.x (mgr.34643) 131 : cluster [DBG] pgmap v118: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:07 smithi153 bash[39117]: cluster 2024-02-10T09:35:06.776652+0000 mgr.x (mgr.34643) 131 : cluster [DBG] pgmap v118: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:10.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:09 smithi153 bash[35780]: cluster 2024-02-10T09:35:08.777398+0000 mgr.x (mgr.34643) 132 : cluster [DBG] pgmap v119: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:10.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:09 smithi153 bash[39117]: cluster 2024-02-10T09:35:08.777398+0000 mgr.x (mgr.34643) 132 : cluster [DBG] pgmap v119: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:09 smithi204 bash[35809]: cluster 2024-02-10T09:35:08.777398+0000 mgr.x (mgr.34643) 132 : cluster [DBG] pgmap v119: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:12.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:11 smithi153 bash[35780]: cluster 2024-02-10T09:35:10.777804+0000 mgr.x (mgr.34643) 133 : cluster [DBG] pgmap v120: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:12.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:11 smithi153 bash[39117]: cluster 2024-02-10T09:35:10.777804+0000 mgr.x (mgr.34643) 133 : cluster [DBG] pgmap v120: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:11 smithi204 bash[35809]: cluster 2024-02-10T09:35:10.777804+0000 mgr.x (mgr.34643) 133 : cluster [DBG] pgmap v120: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:14.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:13 smithi204 bash[35809]: cluster 2024-02-10T09:35:12.778696+0000 mgr.x (mgr.34643) 134 : cluster [DBG] pgmap v121: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:13] "GET /metrics HTTP/1.1" 200 153205 "" "Prometheus/2.33.4" 2024-02-10T09:35:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:13 smithi153 bash[35780]: cluster 2024-02-10T09:35:12.778696+0000 mgr.x (mgr.34643) 134 : cluster [DBG] pgmap v121: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:13 smithi153 bash[39117]: cluster 2024-02-10T09:35:12.778696+0000 mgr.x (mgr.34643) 134 : cluster [DBG] pgmap v121: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:14 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:35:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:15 smithi153 bash[39117]: cluster 2024-02-10T09:35:14.779254+0000 mgr.x (mgr.34643) 135 : cluster [DBG] pgmap v122: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:15 smithi153 bash[35780]: cluster 2024-02-10T09:35:14.779254+0000 mgr.x (mgr.34643) 135 : cluster [DBG] pgmap v122: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:15 smithi204 bash[35809]: cluster 2024-02-10T09:35:14.779254+0000 mgr.x (mgr.34643) 135 : cluster [DBG] pgmap v122: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:17 smithi204 bash[35809]: cluster 2024-02-10T09:35:16.779870+0000 mgr.x (mgr.34643) 136 : cluster [DBG] pgmap v123: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:17 smithi153 bash[35780]: cluster 2024-02-10T09:35:16.779870+0000 mgr.x (mgr.34643) 136 : cluster [DBG] pgmap v123: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:17 smithi153 bash[39117]: cluster 2024-02-10T09:35:16.779870+0000 mgr.x (mgr.34643) 136 : cluster [DBG] pgmap v123: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:19 smithi153 bash[35780]: cluster 2024-02-10T09:35:18.780587+0000 mgr.x (mgr.34643) 137 : cluster [DBG] pgmap v124: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:19 smithi153 bash[39117]: cluster 2024-02-10T09:35:18.780587+0000 mgr.x (mgr.34643) 137 : cluster [DBG] pgmap v124: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:19 smithi204 bash[35809]: cluster 2024-02-10T09:35:18.780587+0000 mgr.x (mgr.34643) 137 : cluster [DBG] pgmap v124: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:21 smithi153 bash[35780]: cluster 2024-02-10T09:35:20.781029+0000 mgr.x (mgr.34643) 138 : cluster [DBG] pgmap v125: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:21 smithi153 bash[39117]: cluster 2024-02-10T09:35:20.781029+0000 mgr.x (mgr.34643) 138 : cluster [DBG] pgmap v125: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:22.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:21 smithi204 bash[35809]: cluster 2024-02-10T09:35:20.781029+0000 mgr.x (mgr.34643) 138 : cluster [DBG] pgmap v125: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.896545+0000 mon.b (mon.2) 214 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:35:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.896843+0000 mon.b (mon.2) 215 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-02-10T09:35:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.897016+0000 mon.b (mon.2) 216 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.897125+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:35:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.897205+0000 mon.b (mon.2) 217 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.897611+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-02-10T09:35:23.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.897823+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[35780]: audit 2024-02-10T09:35:21.898018+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.896545+0000 mon.b (mon.2) 214 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:35:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.896843+0000 mon.b (mon.2) 215 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-02-10T09:35:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.897016+0000 mon.b (mon.2) 216 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.897125+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:35:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.897205+0000 mon.b (mon.2) 217 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.897611+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-02-10T09:35:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.897823+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:22 smithi153 bash[39117]: audit 2024-02-10T09:35:21.898018+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.896545+0000 mon.b (mon.2) 214 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:35:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.896843+0000 mon.b (mon.2) 215 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-02-10T09:35:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.897016+0000 mon.b (mon.2) 216 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.897125+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-10T09:35:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.897205+0000 mon.b (mon.2) 217 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.897611+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-02-10T09:35:23.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.897823+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:35:23.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:22 smithi204 bash[35809]: audit 2024-02-10T09:35:21.898018+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T09:35:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:23 smithi204 bash[35809]: cluster 2024-02-10T09:35:22.781713+0000 mgr.x (mgr.34643) 139 : cluster [DBG] pgmap v126: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:24.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:23] "GET /metrics HTTP/1.1" 200 153206 "" "Prometheus/2.33.4" 2024-02-10T09:35:24.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:23 smithi153 bash[35780]: cluster 2024-02-10T09:35:22.781713+0000 mgr.x (mgr.34643) 139 : cluster [DBG] pgmap v126: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:23 smithi153 bash[39117]: cluster 2024-02-10T09:35:22.781713+0000 mgr.x (mgr.34643) 139 : cluster [DBG] pgmap v126: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:24 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:35:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:25 smithi153 bash[35780]: cluster 2024-02-10T09:35:24.782246+0000 mgr.x (mgr.34643) 140 : cluster [DBG] pgmap v127: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:25 smithi153 bash[39117]: cluster 2024-02-10T09:35:24.782246+0000 mgr.x (mgr.34643) 140 : cluster [DBG] pgmap v127: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:25 smithi204 bash[35809]: cluster 2024-02-10T09:35:24.782246+0000 mgr.x (mgr.34643) 140 : cluster [DBG] pgmap v127: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:27.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:35:27 smithi153 bash[61050]: time="2024-02-10T09:35:27Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:35:28.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:27 smithi204 bash[35809]: cluster 2024-02-10T09:35:26.782722+0000 mgr.x (mgr.34643) 141 : cluster [DBG] pgmap v128: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:27 smithi153 bash[35780]: cluster 2024-02-10T09:35:26.782722+0000 mgr.x (mgr.34643) 141 : cluster [DBG] pgmap v128: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:27 smithi153 bash[39117]: cluster 2024-02-10T09:35:26.782722+0000 mgr.x (mgr.34643) 141 : cluster [DBG] pgmap v128: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:28.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:35:28 smithi204 bash[49655]: time="2024-02-10T09:35:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:35:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:29 smithi153 bash[39117]: cluster 2024-02-10T09:35:28.783582+0000 mgr.x (mgr.34643) 142 : cluster [DBG] pgmap v129: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:29 smithi153 bash[35780]: cluster 2024-02-10T09:35:28.783582+0000 mgr.x (mgr.34643) 142 : cluster [DBG] pgmap v129: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:29 smithi204 bash[35809]: cluster 2024-02-10T09:35:28.783582+0000 mgr.x (mgr.34643) 142 : cluster [DBG] pgmap v129: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:31 smithi153 bash[39117]: cluster 2024-02-10T09:35:30.784074+0000 mgr.x (mgr.34643) 143 : cluster [DBG] pgmap v130: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:31 smithi153 bash[35780]: cluster 2024-02-10T09:35:30.784074+0000 mgr.x (mgr.34643) 143 : cluster [DBG] pgmap v130: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:32.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:31 smithi204 bash[35809]: cluster 2024-02-10T09:35:30.784074+0000 mgr.x (mgr.34643) 143 : cluster [DBG] pgmap v130: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:34.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:33 smithi204 bash[35809]: cluster 2024-02-10T09:35:32.784780+0000 mgr.x (mgr.34643) 144 : cluster [DBG] pgmap v131: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:34.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:33] "GET /metrics HTTP/1.1" 200 153206 "" "Prometheus/2.33.4" 2024-02-10T09:35:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:33 smithi153 bash[35780]: cluster 2024-02-10T09:35:32.784780+0000 mgr.x (mgr.34643) 144 : cluster [DBG] pgmap v131: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:33 smithi153 bash[39117]: cluster 2024-02-10T09:35:32.784780+0000 mgr.x (mgr.34643) 144 : cluster [DBG] pgmap v131: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:35:34.850 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: start operation timed out. Terminating. 2024-02-10T09:35:34.850 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'timeout'. 2024-02-10T09:35:34.850 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:34 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:34.850 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: debug 2024-02-10T09:35:34.619+0000 7ff7ce4fc700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:34.850 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: main() 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:34.851 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:34.852 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: main() 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STOPPING 2024-02-10T09:35:34.853 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STOPPED 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STARTING 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Serving on http://:::9283 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STARTED 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STOPPING 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STOPPED 2024-02-10T09:35:34.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STARTING 2024-02-10T09:35:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.624299+0000 mon.b (mon.2) 218 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T09:35:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.625781+0000 mon.b (mon.2) 219 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T09:35:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.626315+0000 mon.b (mon.2) 220 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]: dispatch 2024-02-10T09:35:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.626788+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]: dispatch 2024-02-10T09:35:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.631208+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]': finished 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.635903+0000 mon.b (mon.2) 221 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.640636+0000 mon.b (mon.2) 222 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.641318+0000 mon.b (mon.2) 223 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.641754+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.645937+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]': finished 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[35780]: audit 2024-02-10T09:35:34.648032+0000 mon.b (mon.2) 224 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.624299+0000 mon.b (mon.2) 218 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.625781+0000 mon.b (mon.2) 219 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.626315+0000 mon.b (mon.2) 220 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.626788+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]: dispatch 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.631208+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]': finished 2024-02-10T09:35:35.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.635903+0000 mon.b (mon.2) 221 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T09:35:35.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.640636+0000 mon.b (mon.2) 222 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T09:35:35.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.641318+0000 mon.b (mon.2) 223 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]: dispatch 2024-02-10T09:35:35.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.641754+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]: dispatch 2024-02-10T09:35:35.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.645937+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]': finished 2024-02-10T09:35:35.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:34 smithi153 bash[39117]: audit 2024-02-10T09:35:34.648032+0000 mon.b (mon.2) 224 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:35:35.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Serving on http://:::9283 2024-02-10T09:35:35.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[44646]: [10/Feb/2024:09:35:34] ENGINE Bus STARTED 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.624299+0000 mon.b (mon.2) 218 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.625781+0000 mon.b (mon.2) 219 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.626315+0000 mon.b (mon.2) 220 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.626788+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.631208+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"http://smithi153.front.sepia.ceph.com:9093","who":"mgr","name":"mgr/dashboard/ALERTMANAGER_API_HOST"}]': finished 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.635903+0000 mon.b (mon.2) 221 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.640636+0000 mon.b (mon.2) 222 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.641318+0000 mon.b (mon.2) 223 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.641754+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]: dispatch 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.645937+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config set","value":"http://smithi204.front.sepia.ceph.com:9095","who":"mgr","name":"mgr/dashboard/PROMETHEUS_API_HOST"}]': finished 2024-02-10T09:35:35.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:34 smithi204 bash[35809]: audit 2024-02-10T09:35:34.648032+0000 mon.b (mon.2) 224 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: cephadm 2024-02-10T09:35:34.620591+0000 mgr.x (mgr.34643) 145 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: main() 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:36.184 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:36.185 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: main() 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:36.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: audit 2024-02-10T09:35:34.624881+0000 mgr.x (mgr.34643) 146 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: audit 2024-02-10T09:35:34.626052+0000 mgr.x (mgr.34643) 147 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: audit 2024-02-10T09:35:34.636400+0000 mgr.x (mgr.34643) 148 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: audit 2024-02-10T09:35:34.640966+0000 mgr.x (mgr.34643) 149 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[39117]: cluster 2024-02-10T09:35:34.785180+0000 mgr.x (mgr.34643) 150 : cluster [DBG] pgmap v132: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: cephadm 2024-02-10T09:35:34.620591+0000 mgr.x (mgr.34643) 145 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.187 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: main() 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:36.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: main() 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: audit 2024-02-10T09:35:34.624881+0000 mgr.x (mgr.34643) 146 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: audit 2024-02-10T09:35:34.626052+0000 mgr.x (mgr.34643) 147 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: audit 2024-02-10T09:35:34.636400+0000 mgr.x (mgr.34643) 148 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: audit 2024-02-10T09:35:34.640966+0000 mgr.x (mgr.34643) 149 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T09:35:36.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:35 smithi153 bash[35780]: cluster 2024-02-10T09:35:34.785180+0000 mgr.x (mgr.34643) 150 : cluster [DBG] pgmap v132: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: cephadm 2024-02-10T09:35:34.620591+0000 mgr.x (mgr.34643) 145 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: main() 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon grafana.a ... 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: main() 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:35:36.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because a timeout was exceeded. 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: audit 2024-02-10T09:35:34.624881+0000 mgr.x (mgr.34643) 146 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: audit 2024-02-10T09:35:34.626052+0000 mgr.x (mgr.34643) 147 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: audit 2024-02-10T09:35:34.636400+0000 mgr.x (mgr.34643) 148 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: audit 2024-02-10T09:35:34.640966+0000 mgr.x (mgr.34643) 149 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T09:35:36.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:35 smithi204 bash[35809]: cluster 2024-02-10T09:35:34.785180+0000 mgr.x (mgr.34643) 150 : cluster [DBG] pgmap v132: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[35780]: audit 2024-02-10T09:35:36.583890+0000 mon.b (mon.2) 225 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]: dispatch 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[35780]: audit 2024-02-10T09:35:36.584601+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]: dispatch 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[35780]: audit 2024-02-10T09:35:36.588393+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]': finished 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[35780]: audit 2024-02-10T09:35:36.590719+0000 mon.b (mon.2) 226 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[35780]: audit 2024-02-10T09:35:36.590976+0000 mon.b (mon.2) 227 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[39117]: audit 2024-02-10T09:35:36.583890+0000 mon.b (mon.2) 225 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]: dispatch 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[39117]: audit 2024-02-10T09:35:36.584601+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]: dispatch 2024-02-10T09:35:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[39117]: audit 2024-02-10T09:35:36.588393+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]': finished 2024-02-10T09:35:37.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[39117]: audit 2024-02-10T09:35:36.590719+0000 mon.b (mon.2) 226 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:37.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:36 smithi153 bash[39117]: audit 2024-02-10T09:35:36.590976+0000 mon.b (mon.2) 227 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:37.259 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:36 smithi204 bash[35809]: audit 2024-02-10T09:35:36.583890+0000 mon.b (mon.2) 225 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]: dispatch 2024-02-10T09:35:37.259 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:36 smithi204 bash[35809]: audit 2024-02-10T09:35:36.584601+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]: dispatch 2024-02-10T09:35:37.259 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:36 smithi204 bash[35809]: audit 2024-02-10T09:35:36.588393+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr.y"}]': finished 2024-02-10T09:35:37.259 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:36 smithi204 bash[35809]: audit 2024-02-10T09:35:36.590719+0000 mon.b (mon.2) 226 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:37.260 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:36 smithi204 bash[35809]: audit 2024-02-10T09:35:36.590976+0000 mon.b (mon.2) 227 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:37.856 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:37 smithi153 bash[35780]: cluster 2024-02-10T09:35:36.785560+0000 mgr.x (mgr.34643) 151 : cluster [DBG] pgmap v133: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:37.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:37 smithi153 bash[39117]: cluster 2024-02-10T09:35:36.785560+0000 mgr.x (mgr.34643) 151 : cluster [DBG] pgmap v133: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:38.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:37 smithi204 bash[35809]: cluster 2024-02-10T09:35:36.785560+0000 mgr.x (mgr.34643) 151 : cluster [DBG] pgmap v133: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:40.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:39 smithi204 bash[35809]: cluster 2024-02-10T09:35:38.786231+0000 mgr.x (mgr.34643) 152 : cluster [DBG] pgmap v134: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:40.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:39 smithi204 bash[35809]: audit 2024-02-10T09:35:39.564920+0000 mon.b (mon.2) 228 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:40.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:39 smithi204 bash[35809]: audit 2024-02-10T09:35:39.565750+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:40.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:39 smithi204 bash[35809]: audit 2024-02-10T09:35:39.568522+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]': finished 2024-02-10T09:35:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[35780]: cluster 2024-02-10T09:35:38.786231+0000 mgr.x (mgr.34643) 152 : cluster [DBG] pgmap v134: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[39117]: cluster 2024-02-10T09:35:38.786231+0000 mgr.x (mgr.34643) 152 : cluster [DBG] pgmap v134: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:40.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[39117]: audit 2024-02-10T09:35:39.564920+0000 mon.b (mon.2) 228 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:40.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[35780]: audit 2024-02-10T09:35:39.564920+0000 mon.b (mon.2) 228 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:40.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[39117]: audit 2024-02-10T09:35:39.565750+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:40.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[39117]: audit 2024-02-10T09:35:39.568522+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]': finished 2024-02-10T09:35:40.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[35780]: audit 2024-02-10T09:35:39.565750+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:40.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:39 smithi153 bash[35780]: audit 2024-02-10T09:35:39.568522+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"03548f93e5ac\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 305869619, \"cpu_percentage\": \"2.54%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563205Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:16:47.651661Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:18:37.161028Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:39.563815Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]': finished 2024-02-10T09:35:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: cluster 2024-02-10T09:35:40.786590+0000 mgr.x (mgr.34643) 153 : cluster [DBG] pgmap v135: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:42.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: cluster 2024-02-10T09:35:40.786590+0000 mgr.x (mgr.34643) 153 : cluster [DBG] pgmap v135: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:42.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.281836+0000 mon.b (mon.2) 229 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:42.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.281836+0000 mon.b (mon.2) 229 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.282638+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.288395+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.569316+0000 mon.b (mon.2) 230 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.570625+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.576322+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.579162+0000 mon.b (mon.2) 231 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.579494+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.580178+0000 mon.b (mon.2) 232 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:35:42.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.580732+0000 mon.b (mon.2) 233 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[39117]: audit 2024-02-10T09:35:41.581535+0000 mon.b (mon.2) 234 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mgr.y","key": "container_image"}]: dispatch 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.282638+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.288395+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.569316+0000 mon.b (mon.2) 230 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.570625+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.576322+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.579162+0000 mon.b (mon.2) 231 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:35:42.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.579494+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:35:42.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.580178+0000 mon.b (mon.2) 232 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:35:42.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.580732+0000 mon.b (mon.2) 233 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:35:42.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:41 smithi153 bash[35780]: audit 2024-02-10T09:35:41.581535+0000 mon.b (mon.2) 234 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mgr.y","key": "container_image"}]: dispatch 2024-02-10T09:35:42.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: cluster 2024-02-10T09:35:40.786590+0000 mgr.x (mgr.34643) 153 : cluster [DBG] pgmap v135: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:42.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.281836+0000 mon.b (mon.2) 229 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.282638+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.288395+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 124256256, \"cpu_percentage\": \"0.85%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279853Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 455920844, \"cpu_percentage\": \"4.06%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.279999Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48643440, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280070Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48779755, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280162Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46483374, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280232Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47133491, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280300Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280366Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 27483176, \"cpu_percentage\": \"0.46%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280407Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35599155, \"cpu_percentage\": \"0.16%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:41.280487Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60104376, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:35:41.280560Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:35:41.280672Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.569316+0000 mon.b (mon.2) 230 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.570625+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.576322+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.579162+0000 mon.b (mon.2) 231 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.579494+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.580178+0000 mon.b (mon.2) 232 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.580732+0000 mon.b (mon.2) 233 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:35:42.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:41 smithi204 bash[35809]: audit 2024-02-10T09:35:41.581535+0000 mon.b (mon.2) 234 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mgr.y","key": "container_image"}]: dispatch 2024-02-10T09:35:43.000 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[39117]: cluster 2024-02-10T09:35:41.568645+0000 mgr.x (mgr.34643) 154 : cluster [DBG] pgmap v136: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:43.000 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[39117]: cluster 2024-02-10T09:35:41.568845+0000 mgr.x (mgr.34643) 155 : cluster [DBG] pgmap v137: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:43.001 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[39117]: cephadm 2024-02-10T09:35:41.581174+0000 mgr.x (mgr.34643) 156 : cephadm [INF] Deploying daemon mgr.y on smithi153 2024-02-10T09:35:43.001 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[39117]: cluster 2024-02-10T09:35:42.286518+0000 mon.a (mon.0) 734 : cluster [WRN] Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:35:43.001 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[35780]: cluster 2024-02-10T09:35:41.568645+0000 mgr.x (mgr.34643) 154 : cluster [DBG] pgmap v136: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:43.001 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[35780]: cluster 2024-02-10T09:35:41.568845+0000 mgr.x (mgr.34643) 155 : cluster [DBG] pgmap v137: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:43.001 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[35780]: cephadm 2024-02-10T09:35:41.581174+0000 mgr.x (mgr.34643) 156 : cephadm [INF] Deploying daemon mgr.y on smithi153 2024-02-10T09:35:43.001 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:42 smithi153 bash[35780]: cluster 2024-02-10T09:35:42.286518+0000 mon.a (mon.0) 734 : cluster [WRN] Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:35:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:42 smithi204 bash[35809]: cluster 2024-02-10T09:35:41.568645+0000 mgr.x (mgr.34643) 154 : cluster [DBG] pgmap v136: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:42 smithi204 bash[35809]: cluster 2024-02-10T09:35:41.568845+0000 mgr.x (mgr.34643) 155 : cluster [DBG] pgmap v137: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:42 smithi204 bash[35809]: cephadm 2024-02-10T09:35:41.581174+0000 mgr.x (mgr.34643) 156 : cephadm [INF] Deploying daemon mgr.y on smithi153 2024-02-10T09:35:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:42 smithi204 bash[35809]: cluster 2024-02-10T09:35:42.286518+0000 mon.a (mon.0) 734 : cluster [WRN] Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:35:43.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:43 smithi153 systemd[1]: Stopping Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:35:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:43] "GET /metrics HTTP/1.1" 200 153208 "" "Prometheus/2.33.4" 2024-02-10T09:35:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:44 smithi153 bash[36014]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:35:45.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:44 smithi153 bash[35780]: cluster 2024-02-10T09:35:43.569502+0000 mgr.x (mgr.34643) 157 : cluster [DBG] pgmap v138: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:44 smithi153 bash[39117]: cluster 2024-02-10T09:35:43.569502+0000 mgr.x (mgr.34643) 157 : cluster [DBG] pgmap v138: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:44 smithi204 bash[35809]: cluster 2024-02-10T09:35:43.569502+0000 mgr.x (mgr.34643) 157 : cluster [DBG] pgmap v138: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:35:45.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:45.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:35:45.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:45.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 bash[50354]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:35:45.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:35:45.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:35:45.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:44 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:47.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:46 smithi153 bash[35780]: cluster 2024-02-10T09:35:45.570063+0000 mgr.x (mgr.34643) 158 : cluster [DBG] pgmap v139: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:47.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:46 smithi153 bash[39117]: cluster 2024-02-10T09:35:45.570063+0000 mgr.x (mgr.34643) 158 : cluster [DBG] pgmap v139: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:47.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:46 smithi204 bash[35809]: cluster 2024-02-10T09:35:45.570063+0000 mgr.x (mgr.34643) 158 : cluster [DBG] pgmap v139: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:49.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:48 smithi153 bash[35780]: cluster 2024-02-10T09:35:47.570473+0000 mgr.x (mgr.34643) 159 : cluster [DBG] pgmap v140: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:49.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:48 smithi153 bash[39117]: cluster 2024-02-10T09:35:47.570473+0000 mgr.x (mgr.34643) 159 : cluster [DBG] pgmap v140: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:49.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:48 smithi204 bash[35809]: cluster 2024-02-10T09:35:47.570473+0000 mgr.x (mgr.34643) 159 : cluster [DBG] pgmap v140: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:50 smithi153 bash[35780]: cluster 2024-02-10T09:35:49.571228+0000 mgr.x (mgr.34643) 160 : cluster [DBG] pgmap v141: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:50 smithi153 bash[39117]: cluster 2024-02-10T09:35:49.571228+0000 mgr.x (mgr.34643) 160 : cluster [DBG] pgmap v141: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:51.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:50 smithi204 bash[35809]: cluster 2024-02-10T09:35:49.571228+0000 mgr.x (mgr.34643) 160 : cluster [DBG] pgmap v141: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:53.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:52 smithi153 bash[35780]: cluster 2024-02-10T09:35:51.571623+0000 mgr.x (mgr.34643) 161 : cluster [DBG] pgmap v142: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:52 smithi153 bash[39117]: cluster 2024-02-10T09:35:51.571623+0000 mgr.x (mgr.34643) 161 : cluster [DBG] pgmap v142: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:52 smithi204 bash[35809]: cluster 2024-02-10T09:35:51.571623+0000 mgr.x (mgr.34643) 161 : cluster [DBG] pgmap v142: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:53.581 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:53 smithi153 bash[61818]: time="2024-02-10T09:35:53Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.y in 10 seconds, resorting to SIGKILL" 2024-02-10T09:35:53.926 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:53 smithi153 bash[61818]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.y 2024-02-10T09:35:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:35:53] "GET /metrics HTTP/1.1" 200 153206 "" "Prometheus/2.33.4" 2024-02-10T09:35:54.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:53 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=137/n/a 2024-02-10T09:35:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:54 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-02-10T09:35:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:54 smithi153 systemd[1]: Stopped Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:54 smithi153 systemd[1]: Starting Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:35:55.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:54 smithi153 bash[35780]: cluster 2024-02-10T09:35:53.572509+0000 mgr.x (mgr.34643) 162 : cluster [DBG] pgmap v143: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:55.152 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:54 smithi153 bash[39117]: cluster 2024-02-10T09:35:53.572509+0000 mgr.x (mgr.34643) 162 : cluster [DBG] pgmap v143: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:55.152 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:55.025+0000 7f11404f4700 1 -- 172.21.15.153:0/2772401578 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b09df724e0 con 0x55b09df46c00 2024-02-10T09:35:55.152 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[62015]: a9598abcef4876b02d6ff22bef70c377fc29df3effa5f1a2de6fb2800cf04f0c 2024-02-10T09:35:55.152 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:55 smithi153 systemd[1]: Started Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:54 smithi204 bash[35809]: cluster 2024-02-10T09:35:53.572509+0000 mgr.x (mgr.34643) 162 : cluster [DBG] pgmap v143: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:55.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:35:55.206 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:35:55.206 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:55.206 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:35:55.206 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:55.206 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:35:55.206 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:55.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:55.150+0000 7f1150af0680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:35:55.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:55.237+0000 7f1150af0680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:35:55.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[50372]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:35:55.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:35:55.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:35:55.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:55 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:55.900 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:55.601+0000 7f1150af0680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: audit 2024-02-10T09:35:55.170098+0000 mon.b (mon.2) 235 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: audit 2024-02-10T09:35:55.170803+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: audit 2024-02-10T09:35:55.173775+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]': finished 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: audit 2024-02-10T09:35:55.175962+0000 mon.b (mon.2) 236 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: audit 2024-02-10T09:35:55.176594+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: audit 2024-02-10T09:35:55.181121+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: cephadm 2024-02-10T09:35:55.183683+0000 mgr.x (mgr.34643) 163 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:35:55.941 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:55 smithi204 bash[35809]: cephadm 2024-02-10T09:35:55.189936+0000 mgr.x (mgr.34643) 164 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:35:56.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: audit 2024-02-10T09:35:55.170098+0000 mon.b (mon.2) 235 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:56.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: audit 2024-02-10T09:35:55.170098+0000 mon.b (mon.2) 235 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: audit 2024-02-10T09:35:55.170803+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:56.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: audit 2024-02-10T09:35:55.170803+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]: dispatch 2024-02-10T09:35:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: audit 2024-02-10T09:35:55.173775+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]': finished 2024-02-10T09:35:56.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: audit 2024-02-10T09:35:55.173775+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {\"mgr.y\": \"redeploy\"}}"}]': finished 2024-02-10T09:35:56.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: audit 2024-02-10T09:35:55.175962+0000 mon.b (mon.2) 236 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:56.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: audit 2024-02-10T09:35:55.175962+0000 mon.b (mon.2) 236 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:56.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: audit 2024-02-10T09:35:55.176594+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:56.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: audit 2024-02-10T09:35:55.176594+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:35:56.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: audit 2024-02-10T09:35:55.181121+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:35:56.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: cephadm 2024-02-10T09:35:55.183683+0000 mgr.x (mgr.34643) 163 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:35:56.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[35780]: cephadm 2024-02-10T09:35:55.189936+0000 mgr.x (mgr.34643) 164 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:35:56.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: audit 2024-02-10T09:35:55.181121+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 108737331, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563051Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.70%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563277Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44889538, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563342Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44533022, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563407Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42897244, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563469Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46776975, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563533Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563595Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34393292, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:35:39.563637Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.11%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:35:39.563702Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:35:56.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: cephadm 2024-02-10T09:35:55.183683+0000 mgr.x (mgr.34643) 163 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:35:56.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:55 smithi153 bash[39117]: cephadm 2024-02-10T09:35:55.189936+0000 mgr.x (mgr.34643) 164 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:35:56.650 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:56.353+0000 7f1150af0680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:35:56.650 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:56.436+0000 7f1150af0680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:35:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:57 smithi153 bash[35780]: cluster 2024-02-10T09:35:55.572952+0000 mgr.x (mgr.34643) 165 : cluster [DBG] pgmap v144: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:57.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:56.649+0000 7f1150af0680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:35:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:57 smithi153 bash[39117]: cluster 2024-02-10T09:35:55.572952+0000 mgr.x (mgr.34643) 165 : cluster [DBG] pgmap v144: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:57.526 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[35809]: cluster 2024-02-10T09:35:55.572952+0000 mgr.x (mgr.34643) 165 : cluster [DBG] pgmap v144: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:57.776 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:57.776 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:35:57.776 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:57.776 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:35:57.776 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:57.777 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:35:57.777 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:57.903 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:57.604+0000 7f1150af0680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:35:57.903 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:57.732+0000 7f1150af0680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:35:57.903 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:57.812+0000 7f1150af0680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:35:57.903 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:57.902+0000 7f1150af0680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:35:58.081 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:35:58.081 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[50569]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:35:58.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:35:57 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: debug 2024-02-10T09:35:57.925+0000 7ff7ce4fc700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: main() 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:35:58.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:58.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: main() 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:58.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:58.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:58.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:35:57 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:58.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:58.109+0000 7f1150af0680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:35:58.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:58 smithi153 bash[39117]: audit 2024-02-10T09:35:57.927535+0000 mon.b (mon.2) 237 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:35:58.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:58 smithi153 bash[39117]: audit 2024-02-10T09:35:57.928828+0000 mon.b (mon.2) 238 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:58.532 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:58.430+0000 7f1150af0680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:35:58.532 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:58 smithi153 bash[35780]: audit 2024-02-10T09:35:57.927535+0000 mon.b (mon.2) 237 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:35:58.532 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:58 smithi153 bash[35780]: audit 2024-02-10T09:35:57.928828+0000 mon.b (mon.2) 238 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:58.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:58 smithi204 bash[35809]: audit 2024-02-10T09:35:57.927535+0000 mon.b (mon.2) 237 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:35:58.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:58 smithi204 bash[35809]: audit 2024-02-10T09:35:57.928828+0000 mon.b (mon.2) 238 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:35:58.801 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:58.531+0000 7f1150af0680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: cluster 2024-02-10T09:35:57.573405+0000 mgr.x (mgr.34643) 166 : cluster [DBG] pgmap v145: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: cephadm 2024-02-10T09:35:57.926417+0000 mgr.x (mgr.34643) 167 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.491 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: main() 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.492 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: cluster 2024-02-10T09:35:57.573405+0000 mgr.x (mgr.34643) 166 : cluster [DBG] pgmap v145: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: cephadm 2024-02-10T09:35:57.926417+0000 mgr.x (mgr.34643) 167 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:59.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: main() 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:59.494 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:35:59.495 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: main() 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:59.496 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:59.300+0000 7f1150af0680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:59.390+0000 7f1150af0680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:59.489+0000 7f1150af0680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.497 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: main() 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.498 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:35:59 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: cluster 2024-02-10T09:35:57.573405+0000 mgr.x (mgr.34643) 166 : cluster [DBG] pgmap v145: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:35:59.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: cephadm 2024-02-10T09:35:57.926417+0000 mgr.x (mgr.34643) 167 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: main() 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:59.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: main() 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:35:59.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:35:59 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:35:59.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:59.672+0000 7f1150af0680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:36:00.267 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:35:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:35:59.961+0000 7f1150af0680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:36:00.268 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:36:00.111+0000 7f1150af0680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:36:00.557 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:36:00.266+0000 7f1150af0680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:36:00.557 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:36:00.464+0000 7f1150af0680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:36:00.557 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:36:00.556+0000 7f1150af0680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:36:00.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:00] ENGINE Bus STARTING 2024-02-10T09:36:00.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: CherryPy Checker: 2024-02-10T09:36:00.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: The Application mounted at '' has an empty config. 2024-02-10T09:36:00.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:36:00.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:00] ENGINE Serving on http://:::9283 2024-02-10T09:36:00.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:00] ENGINE Bus STARTED 2024-02-10T09:36:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: cluster 2024-02-10T09:35:59.574274+0000 mgr.x (mgr.34643) 168 : cluster [DBG] pgmap v146: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: cluster 2024-02-10T09:36:00.569650+0000 mon.a (mon.0) 739 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:36:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: cluster 2024-02-10T09:36:00.569769+0000 mon.a (mon.0) 740 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:36:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: audit 2024-02-10T09:36:00.571863+0000 mon.a (mon.0) 741 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T09:36:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: audit 2024-02-10T09:36:00.572223+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: audit 2024-02-10T09:36:00.574296+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T09:36:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:01 smithi204 bash[35809]: audit 2024-02-10T09:36:00.575265+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: cluster 2024-02-10T09:35:59.574274+0000 mgr.x (mgr.34643) 168 : cluster [DBG] pgmap v146: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: cluster 2024-02-10T09:36:00.569650+0000 mon.a (mon.0) 739 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: cluster 2024-02-10T09:36:00.569769+0000 mon.a (mon.0) 740 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: audit 2024-02-10T09:36:00.571863+0000 mon.a (mon.0) 741 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: audit 2024-02-10T09:36:00.572223+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: audit 2024-02-10T09:36:00.574296+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[39117]: audit 2024-02-10T09:36:00.575265+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: cluster 2024-02-10T09:35:59.574274+0000 mgr.x (mgr.34643) 168 : cluster [DBG] pgmap v146: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: cluster 2024-02-10T09:36:00.569650+0000 mon.a (mon.0) 739 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:36:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: cluster 2024-02-10T09:36:00.569769+0000 mon.a (mon.0) 740 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:36:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: audit 2024-02-10T09:36:00.571863+0000 mon.a (mon.0) 741 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T09:36:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: audit 2024-02-10T09:36:00.572223+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: audit 2024-02-10T09:36:00.574296+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T09:36:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:01 smithi153 bash[35780]: audit 2024-02-10T09:36:00.575265+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:02.484 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: cluster 2024-02-10T09:36:01.207225+0000 mon.a (mon.0) 745 : cluster [DBG] mgrmap e26: x(active, since 4m), standbys: y 2024-02-10T09:36:02.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: audit 2024-02-10T09:36:01.636476+0000 mon.b (mon.2) 239 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:36:02.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: audit 2024-02-10T09:36:01.637215+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:36:02.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: audit 2024-02-10T09:36:01.640390+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:36:02.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: audit 2024-02-10T09:36:01.643177+0000 mon.b (mon.2) 240 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:02.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: audit 2024-02-10T09:36:01.643546+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:02.485 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:02 smithi204 bash[35809]: audit 2024-02-10T09:36:01.645684+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:02.591 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: cluster 2024-02-10T09:36:01.207225+0000 mon.a (mon.0) 745 : cluster [DBG] mgrmap e26: x(active, since 4m), standbys: y 2024-02-10T09:36:02.592 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: cluster 2024-02-10T09:36:01.207225+0000 mon.a (mon.0) 745 : cluster [DBG] mgrmap e26: x(active, since 4m), standbys: y 2024-02-10T09:36:02.592 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: audit 2024-02-10T09:36:01.636476+0000 mon.b (mon.2) 239 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:36:02.592 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: audit 2024-02-10T09:36:01.636476+0000 mon.b (mon.2) 239 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:36:02.593 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: audit 2024-02-10T09:36:01.637215+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:36:02.593 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: audit 2024-02-10T09:36:01.637215+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]: dispatch 2024-02-10T09:36:02.593 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: audit 2024-02-10T09:36:01.640390+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:36:02.593 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: audit 2024-02-10T09:36:01.643177+0000 mon.b (mon.2) 240 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:02.594 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: audit 2024-02-10T09:36:01.643546+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:02.594 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[39117]: audit 2024-02-10T09:36:01.645684+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:02.594 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: audit 2024-02-10T09:36:01.640390+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 113875353, \"cpu_percentage\": \"1.27%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634277Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 242640486, \"cpu_percentage\": \"116.03%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:01.634474Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 123417395, \"cpu_percentage\": \"0.69%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634562Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44931481, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634641Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44585451, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634718Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 42928701, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634795Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46871347, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634873Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634949Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 34477178, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:01.634997Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27231518, \"cpu_percentage\": \"0.12%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:01.635094Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:01.635255Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}}"}]': finished 2024-02-10T09:36:02.594 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: audit 2024-02-10T09:36:01.643177+0000 mon.b (mon.2) 240 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:02.594 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: audit 2024-02-10T09:36:01.643546+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:02.594 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:02 smithi153 bash[35780]: audit 2024-02-10T09:36:01.645684+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:03 smithi204 bash[35809]: cluster 2024-02-10T09:36:01.574809+0000 mgr.x (mgr.34643) 169 : cluster [DBG] pgmap v147: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:03 smithi204 bash[35809]: cluster 2024-02-10T09:36:01.642205+0000 mgr.x (mgr.34643) 170 : cluster [DBG] pgmap v148: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:03 smithi204 bash[35809]: cluster 2024-02-10T09:36:01.642421+0000 mgr.x (mgr.34643) 171 : cluster [DBG] pgmap v149: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:03 smithi204 bash[35809]: cephadm 2024-02-10T09:36:01.653901+0000 mgr.x (mgr.34643) 172 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:03 smithi204 bash[35809]: cephadm 2024-02-10T09:36:01.660536+0000 mgr.x (mgr.34643) 173 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[39117]: cluster 2024-02-10T09:36:01.574809+0000 mgr.x (mgr.34643) 169 : cluster [DBG] pgmap v147: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[39117]: cluster 2024-02-10T09:36:01.642205+0000 mgr.x (mgr.34643) 170 : cluster [DBG] pgmap v148: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[39117]: cluster 2024-02-10T09:36:01.642421+0000 mgr.x (mgr.34643) 171 : cluster [DBG] pgmap v149: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[39117]: cephadm 2024-02-10T09:36:01.653901+0000 mgr.x (mgr.34643) 172 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[39117]: cephadm 2024-02-10T09:36:01.660536+0000 mgr.x (mgr.34643) 173 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:03.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[35780]: cluster 2024-02-10T09:36:01.574809+0000 mgr.x (mgr.34643) 169 : cluster [DBG] pgmap v147: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[35780]: cluster 2024-02-10T09:36:01.642205+0000 mgr.x (mgr.34643) 170 : cluster [DBG] pgmap v148: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[35780]: cluster 2024-02-10T09:36:01.642421+0000 mgr.x (mgr.34643) 171 : cluster [DBG] pgmap v149: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:03.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[35780]: cephadm 2024-02-10T09:36:01.653901+0000 mgr.x (mgr.34643) 172 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:03.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:03 smithi153 bash[35780]: cephadm 2024-02-10T09:36:01.660536+0000 mgr.x (mgr.34643) 173 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:03.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:03] "GET /metrics HTTP/1.1" 200 153206 "" "Prometheus/2.33.4" 2024-02-10T09:36:04.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[35809]: cluster 2024-02-10T09:36:03.643265+0000 mgr.x (mgr.34643) 174 : cluster [DBG] pgmap v150: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:04.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:04.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:04.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[50778]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:04.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:04.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:04.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:04 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:04.583 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: debug 2024-02-10T09:36:04.295+0000 7ff7ce4fc700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:04.583 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:04.583 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:36:04.583 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: main() 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:04.584 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:36:04.585 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: main() 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:04.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:04 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:04.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:04 smithi153 bash[35780]: cluster 2024-02-10T09:36:03.643265+0000 mgr.x (mgr.34643) 174 : cluster [DBG] pgmap v150: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:36:04.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:04 smithi153 bash[39117]: cluster 2024-02-10T09:36:03.643265+0000 mgr.x (mgr.34643) 174 : cluster [DBG] pgmap v150: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:04.690 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:04 smithi153 bash[61050]: time="2024-02-10T09:36:04Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: cephadm 2024-02-10T09:36:04.297117+0000 mgr.x (mgr.34643) 175 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:05.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: main() 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:36:05.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:05.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: main() 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: cephadm 2024-02-10T09:36:04.297117+0000 mgr.x (mgr.34643) 175 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: main() 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:05.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: cephadm 2024-02-10T09:36:04.297117+0000 mgr.x (mgr.34643) 175 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: main() 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:36:05.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:05.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: main() 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:36:05.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:05.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: main() 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:05.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:05 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:06.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:05 smithi204 bash[49655]: time="2024-02-10T09:36:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:36:06.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:06 smithi204 bash[35809]: cluster 2024-02-10T09:36:05.643688+0000 mgr.x (mgr.34643) 176 : cluster [DBG] pgmap v151: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:06.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:06 smithi153 bash[35780]: cluster 2024-02-10T09:36:05.643688+0000 mgr.x (mgr.34643) 176 : cluster [DBG] pgmap v151: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:06.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:06 smithi153 bash[39117]: cluster 2024-02-10T09:36:05.643688+0000 mgr.x (mgr.34643) 176 : cluster [DBG] pgmap v151: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:09.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:08 smithi204 bash[35809]: cluster 2024-02-10T09:36:07.644082+0000 mgr.x (mgr.34643) 177 : cluster [DBG] pgmap v152: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:08 smithi153 bash[35780]: cluster 2024-02-10T09:36:07.644082+0000 mgr.x (mgr.34643) 177 : cluster [DBG] pgmap v152: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:08 smithi153 bash[39117]: cluster 2024-02-10T09:36:07.644082+0000 mgr.x (mgr.34643) 177 : cluster [DBG] pgmap v152: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:11.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:10 smithi204 bash[35809]: cluster 2024-02-10T09:36:09.644761+0000 mgr.x (mgr.34643) 178 : cluster [DBG] pgmap v153: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:10 smithi153 bash[35780]: cluster 2024-02-10T09:36:09.644761+0000 mgr.x (mgr.34643) 178 : cluster [DBG] pgmap v153: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:11.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:10 smithi153 bash[39117]: cluster 2024-02-10T09:36:09.644761+0000 mgr.x (mgr.34643) 178 : cluster [DBG] pgmap v153: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:13.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:12 smithi204 bash[35809]: cluster 2024-02-10T09:36:11.645199+0000 mgr.x (mgr.34643) 179 : cluster [DBG] pgmap v154: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:13.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:12 smithi153 bash[35780]: cluster 2024-02-10T09:36:11.645199+0000 mgr.x (mgr.34643) 179 : cluster [DBG] pgmap v154: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:13.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:12 smithi153 bash[39117]: cluster 2024-02-10T09:36:11.645199+0000 mgr.x (mgr.34643) 179 : cluster [DBG] pgmap v154: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:13] "GET /metrics HTTP/1.1" 200 153209 "" "Prometheus/2.33.4" 2024-02-10T09:36:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:36:14.802 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:36:14.802 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:36:14.802 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:14.802 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 bash[50796]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:14.803 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:14.804 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:14.804 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:14 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:15.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:14 smithi204 bash[35809]: cluster 2024-02-10T09:36:13.645861+0000 mgr.x (mgr.34643) 180 : cluster [DBG] pgmap v155: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:14 smithi153 bash[35780]: cluster 2024-02-10T09:36:13.645861+0000 mgr.x (mgr.34643) 180 : cluster [DBG] pgmap v155: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:14 smithi153 bash[39117]: cluster 2024-02-10T09:36:13.645861+0000 mgr.x (mgr.34643) 180 : cluster [DBG] pgmap v155: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:16.813 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:16 smithi153 bash[35780]: cluster 2024-02-10T09:36:15.646305+0000 mgr.x (mgr.34643) 181 : cluster [DBG] pgmap v156: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:17.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:16 smithi204 bash[35809]: cluster 2024-02-10T09:36:15.646305+0000 mgr.x (mgr.34643) 181 : cluster [DBG] pgmap v156: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:17.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:16 smithi153 bash[39117]: cluster 2024-02-10T09:36:15.646305+0000 mgr.x (mgr.34643) 181 : cluster [DBG] pgmap v156: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:19.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:18 smithi204 bash[35809]: cluster 2024-02-10T09:36:17.646729+0000 mgr.x (mgr.34643) 182 : cluster [DBG] pgmap v157: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:19.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:18 smithi153 bash[39117]: cluster 2024-02-10T09:36:17.646729+0000 mgr.x (mgr.34643) 182 : cluster [DBG] pgmap v157: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:19.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:18 smithi153 bash[35780]: cluster 2024-02-10T09:36:17.646729+0000 mgr.x (mgr.34643) 182 : cluster [DBG] pgmap v157: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:21.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:20 smithi204 bash[35809]: cluster 2024-02-10T09:36:19.647388+0000 mgr.x (mgr.34643) 183 : cluster [DBG] pgmap v158: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:20 smithi153 bash[35780]: cluster 2024-02-10T09:36:19.647388+0000 mgr.x (mgr.34643) 183 : cluster [DBG] pgmap v158: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:21.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:20 smithi153 bash[39117]: cluster 2024-02-10T09:36:19.647388+0000 mgr.x (mgr.34643) 183 : cluster [DBG] pgmap v158: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:23.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: cluster 2024-02-10T09:36:21.647913+0000 mgr.x (mgr.34643) 184 : cluster [DBG] pgmap v159: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.900382+0000 mon.b (mon.2) 241 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.900636+0000 mon.b (mon.2) 242 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.900871+0000 mon.b (mon.2) 243 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.901095+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.901153+0000 mon.b (mon.2) 244 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.901781+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.902033+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:36:23.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:22 smithi204 bash[35809]: audit 2024-02-10T09:36:21.902285+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: cluster 2024-02-10T09:36:21.647913+0000 mgr.x (mgr.34643) 184 : cluster [DBG] pgmap v159: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.900382+0000 mon.b (mon.2) 241 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.900636+0000 mon.b (mon.2) 242 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.900871+0000 mon.b (mon.2) 243 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.901095+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.901153+0000 mon.b (mon.2) 244 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.901781+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.902033+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[35780]: audit 2024-02-10T09:36:21.902285+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: cluster 2024-02-10T09:36:21.647913+0000 mgr.x (mgr.34643) 184 : cluster [DBG] pgmap v159: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.900382+0000 mon.b (mon.2) 241 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.900636+0000 mon.b (mon.2) 242 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.900871+0000 mon.b (mon.2) 243 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.901095+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.901153+0000 mon.b (mon.2) 244 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.901781+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.902033+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:36:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:22 smithi153 bash[39117]: audit 2024-02-10T09:36:21.902285+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:36:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:23] "GET /metrics HTTP/1.1" 200 153198 "" "Prometheus/2.33.4" 2024-02-10T09:36:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:36:24.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:36:24.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:36:24.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:24.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:24.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 bash[50813]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:24.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:24.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:24.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:24 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:24 smithi153 bash[35780]: cluster 2024-02-10T09:36:23.648800+0000 mgr.x (mgr.34643) 185 : cluster [DBG] pgmap v160: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:24 smithi153 bash[39117]: cluster 2024-02-10T09:36:23.648800+0000 mgr.x (mgr.34643) 185 : cluster [DBG] pgmap v160: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:25.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:24 smithi204 bash[35809]: cluster 2024-02-10T09:36:23.648800+0000 mgr.x (mgr.34643) 185 : cluster [DBG] pgmap v160: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:27.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:26 smithi153 bash[35780]: cluster 2024-02-10T09:36:25.649250+0000 mgr.x (mgr.34643) 186 : cluster [DBG] pgmap v161: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:26 smithi153 bash[39117]: cluster 2024-02-10T09:36:25.649250+0000 mgr.x (mgr.34643) 186 : cluster [DBG] pgmap v161: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:27.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:26 smithi204 bash[35809]: cluster 2024-02-10T09:36:25.649250+0000 mgr.x (mgr.34643) 186 : cluster [DBG] pgmap v161: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:28.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:36:29.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:28 smithi153 bash[35780]: cluster 2024-02-10T09:36:27.649669+0000 mgr.x (mgr.34643) 187 : cluster [DBG] pgmap v162: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:29.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:28 smithi153 bash[39117]: cluster 2024-02-10T09:36:27.649669+0000 mgr.x (mgr.34643) 187 : cluster [DBG] pgmap v162: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:28 smithi204 bash[35809]: cluster 2024-02-10T09:36:27.649669+0000 mgr.x (mgr.34643) 187 : cluster [DBG] pgmap v162: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (4m) 29s ago 5m 25.9M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 error 49s ago 7m 26.2M - 6.6.2 87a51ecf0b1c 72f83803a230 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (9m) 49s ago 17m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc 46c6073a79be 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (36s) 29s ago 19m 231M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (19m) 29s ago 19m 108M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (18m) 49s ago 18m 118M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:36:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (18m) 29s ago 18m 117M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 error 29s ago 10m - - 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 error 49s ago 10m - - 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (17m) 29s ago 17m 42.8M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (16m) 29s ago 17m 42.5M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (16m) 29s ago 16m 40.9M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (16m) 29s ago 16m 44.7M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (16m) 49s ago 16m 46.3M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (15m) 49s ago 16m 46.5M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (15m) 49s ago 15m 44.3M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (15m) 49s ago 15m 44.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (4m) 49s ago 4m 57.3M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (7m) 29s ago 7m 32.8M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:36:30.785 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (7m) 49s ago 7m 33.9M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:36:31.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: cluster 2024-02-10T09:36:29.650559+0000 mgr.x (mgr.34643) 188 : cluster [DBG] pgmap v163: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.772840+0000 mon.b (mon.2) 245 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.773305+0000 mon.b (mon.2) 246 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.773624+0000 mon.b (mon.2) 247 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.773959+0000 mon.b (mon.2) 248 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.774286+0000 mon.b (mon.2) 249 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.774626+0000 mon.b (mon.2) 250 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.776663+0000 mon.b (mon.2) 251 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.777017+0000 mon.b (mon.2) 252 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.777358+0000 mon.b (mon.2) 253 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.777687+0000 mon.b (mon.2) 254 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.175 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[39117]: audit 2024-02-10T09:36:30.778010+0000 mon.b (mon.2) 255 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: cluster 2024-02-10T09:36:29.650559+0000 mgr.x (mgr.34643) 188 : cluster [DBG] pgmap v163: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.772840+0000 mon.b (mon.2) 245 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.773305+0000 mon.b (mon.2) 246 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.773624+0000 mon.b (mon.2) 247 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.773959+0000 mon.b (mon.2) 248 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.774286+0000 mon.b (mon.2) 249 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.774626+0000 mon.b (mon.2) 250 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.776663+0000 mon.b (mon.2) 251 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.777017+0000 mon.b (mon.2) 252 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.777358+0000 mon.b (mon.2) 253 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.777687+0000 mon.b (mon.2) 254 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:30 smithi153 bash[35780]: audit 2024-02-10T09:36:30.778010+0000 mon.b (mon.2) 255 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: cluster 2024-02-10T09:36:29.650559+0000 mgr.x (mgr.34643) 188 : cluster [DBG] pgmap v163: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.772840+0000 mon.b (mon.2) 245 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.773305+0000 mon.b (mon.2) 246 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.773624+0000 mon.b (mon.2) 247 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.773959+0000 mon.b (mon.2) 248 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.774286+0000 mon.b (mon.2) 249 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.774626+0000 mon.b (mon.2) 250 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.776663+0000 mon.b (mon.2) 251 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.777017+0000 mon.b (mon.2) 252 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.777358+0000 mon.b (mon.2) 253 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.777687+0000 mon.b (mon.2) 254 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:30 smithi204 bash[35809]: audit 2024-02-10T09:36:30.778010+0000 mon.b (mon.2) 255 : audit [DBG] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:31.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:36:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:31 smithi153 bash[35780]: audit 2024-02-10T09:36:30.770483+0000 mgr.x (mgr.34643) 189 : audit [DBG] from='client.24778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:36:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:31 smithi153 bash[39117]: audit 2024-02-10T09:36:30.770483+0000 mgr.x (mgr.34643) 189 : audit [DBG] from='client.24778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:36:32.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:31 smithi204 bash[35809]: audit 2024-02-10T09:36:30.770483+0000 mgr.x (mgr.34643) 189 : audit [DBG] from='client.24778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:36:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:32 smithi153 bash[35780]: cluster 2024-02-10T09:36:31.651077+0000 mgr.x (mgr.34643) 190 : cluster [DBG] pgmap v164: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:32.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:32 smithi153 bash[39117]: cluster 2024-02-10T09:36:31.651077+0000 mgr.x (mgr.34643) 190 : cluster [DBG] pgmap v164: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:33.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:32 smithi204 bash[35809]: cluster 2024-02-10T09:36:31.651077+0000 mgr.x (mgr.34643) 190 : cluster [DBG] pgmap v164: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:33.393 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:36:33.394 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:36:33.395 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:36:33.873 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:33] "GET /metrics HTTP/1.1" 200 153198 "" "Prometheus/2.33.4" 2024-02-10T09:36:33.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph -s' 2024-02-10T09:36:33.996 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:33 smithi153 bash[39117]: audit 2024-02-10T09:36:33.393433+0000 mon.a (mon.0) 754 : audit [DBG] from='client.? 172.21.15.153:0/1084428522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:36:33.996 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:33 smithi153 bash[35780]: audit 2024-02-10T09:36:33.393433+0000 mon.a (mon.0) 754 : audit [DBG] from='client.? 172.21.15.153:0/1084428522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:36:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:33 smithi204 bash[35809]: audit 2024-02-10T09:36:33.393433+0000 mon.a (mon.0) 754 : audit [DBG] from='client.? 172.21.15.153:0/1084428522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:36:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:36:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:34 smithi153 bash[39117]: cluster 2024-02-10T09:36:33.651782+0000 mgr.x (mgr.34643) 191 : cluster [DBG] pgmap v165: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:34 smithi153 bash[35780]: cluster 2024-02-10T09:36:33.651782+0000 mgr.x (mgr.34643) 191 : cluster [DBG] pgmap v165: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:34 smithi204 bash[35809]: cluster 2024-02-10T09:36:33.651782+0000 mgr.x (mgr.34643) 191 : cluster [DBG] pgmap v165: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:35.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 bash[50830]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:35.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:34 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:35.900 INFO:teuthology.orchestra.run.smithi153.stdout: cluster: 2024-02-10T09:36:35.900 INFO:teuthology.orchestra.run.smithi153.stdout: id: 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: health: HEALTH_WARN 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: 3 failed cephadm daemon(s) 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: services: 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: mgr: x(active, since 5m), standbys: y 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: rgw: 2 daemons active (r.z.smithi153.zyveqt, r.z.smithi204.kmhqqg) 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: task status: 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: data: 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: pools: 5 pools, 129 pgs 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: objects: 202 objects, 7.0 KiB 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: usage: 8.0 GiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:35.901 INFO:teuthology.orchestra.run.smithi153.stdout: pgs: 129 active+clean 2024-02-10T09:36:35.902 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:36:36.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph mgr fail' 2024-02-10T09:36:36.885 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:36 smithi153 bash[35780]: cluster 2024-02-10T09:36:35.652261+0000 mgr.x (mgr.34643) 192 : cluster [DBG] pgmap v166: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:36.885 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:36 smithi153 bash[35780]: audit 2024-02-10T09:36:35.899968+0000 mon.a (mon.0) 755 : audit [DBG] from='client.? 172.21.15.153:0/2091011599' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:36:37.157 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:36 smithi153 bash[39117]: cluster 2024-02-10T09:36:35.652261+0000 mgr.x (mgr.34643) 192 : cluster [DBG] pgmap v166: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:37.157 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:36 smithi153 bash[39117]: audit 2024-02-10T09:36:35.899968+0000 mon.a (mon.0) 755 : audit [DBG] from='client.? 172.21.15.153:0/2091011599' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:36:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:36 smithi204 bash[35809]: cluster 2024-02-10T09:36:35.652261+0000 mgr.x (mgr.34643) 192 : cluster [DBG] pgmap v166: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:36 smithi204 bash[35809]: audit 2024-02-10T09:36:35.899968+0000 mon.a (mon.0) 755 : audit [DBG] from='client.? 172.21.15.153:0/2091011599' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: cluster 2024-02-10T09:36:37.652668+0000 mgr.x (mgr.34643) 193 : cluster [DBG] pgmap v167: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: audit 2024-02-10T09:36:38.535605+0000 mon.a (mon.0) 756 : audit [INF] from='client.? 172.21.15.153:0/1598783978' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: cluster 2024-02-10T09:36:38.540509+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-10T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: audit 2024-02-10T09:36:38.540560+0000 mon.b (mon.2) 256 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: audit 2024-02-10T09:36:38.541282+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: audit 2024-02-10T09:36:38.542024+0000 mon.b (mon.2) 257 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[39117]: audit 2024-02-10T09:36:38.542353+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: cluster 2024-02-10T09:36:37.652668+0000 mgr.x (mgr.34643) 193 : cluster [DBG] pgmap v167: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: audit 2024-02-10T09:36:38.535605+0000 mon.a (mon.0) 756 : audit [INF] from='client.? 172.21.15.153:0/1598783978' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: cluster 2024-02-10T09:36:38.540509+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: audit 2024-02-10T09:36:38.540560+0000 mon.b (mon.2) 256 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: audit 2024-02-10T09:36:38.541282+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: audit 2024-02-10T09:36:38.542024+0000 mon.b (mon.2) 257 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:36:39.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:38 smithi153 bash[35780]: audit 2024-02-10T09:36:38.542353+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:36:39.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[44646]: debug 2024-02-10T09:36:38.895+0000 7ff8358a1700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[44646]: ignoring --setuser ceph since I am not root 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[44646]: ignoring --setgroup ceph since I am not root 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[44646]: debug 2024-02-10T09:36:38.938+0000 7fc28ea43700 1 -- 172.21.15.204:0/2770333831 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578514c04e0 con 0x557851496c00 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[44646]: debug 2024-02-10T09:36:39.059+0000 7fc29ec21680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[44646]: debug 2024-02-10T09:36:39.140+0000 7fc29ec21680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: cluster 2024-02-10T09:36:37.652668+0000 mgr.x (mgr.34643) 193 : cluster [DBG] pgmap v167: 129 pgs: 129 active+clean; 7.0 KiB data, 5.7 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: audit 2024-02-10T09:36:38.535605+0000 mon.a (mon.0) 756 : audit [INF] from='client.? 172.21.15.153:0/1598783978' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: cluster 2024-02-10T09:36:38.540509+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-10T09:36:39.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: audit 2024-02-10T09:36:38.540560+0000 mon.b (mon.2) 256 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:36:39.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: audit 2024-02-10T09:36:38.541282+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:36:39.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: audit 2024-02-10T09:36:38.542024+0000 mon.b (mon.2) 257 : audit [INF] from='mgr.34643 172.21.15.204:0/3493872294' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:36:39.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:38 smithi204 bash[35809]: audit 2024-02-10T09:36:38.542353+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:36:39.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 180' 2024-02-10T09:36:39.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[44646]: debug 2024-02-10T09:36:39.502+0000 7fc29ec21680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:36:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[35780]: audit 2024-02-10T09:36:38.896057+0000 mon.a (mon.0) 760 : audit [INF] from='client.? 172.21.15.153:0/1598783978' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:36:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[35780]: cluster 2024-02-10T09:36:38.896104+0000 mon.a (mon.0) 761 : cluster [DBG] mgrmap e27: y(active, starting, since 0.360088s) 2024-02-10T09:36:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[35780]: audit 2024-02-10T09:36:39.543049+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:36:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[35780]: cluster 2024-02-10T09:36:39.543093+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-10T09:36:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[39117]: audit 2024-02-10T09:36:38.896057+0000 mon.a (mon.0) 760 : audit [INF] from='client.? 172.21.15.153:0/1598783978' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:36:40.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[39117]: cluster 2024-02-10T09:36:38.896104+0000 mon.a (mon.0) 761 : cluster [DBG] mgrmap e27: y(active, starting, since 0.360088s) 2024-02-10T09:36:40.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[39117]: audit 2024-02-10T09:36:39.543049+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:36:40.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:39 smithi153 bash[39117]: cluster 2024-02-10T09:36:39.543093+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-10T09:36:40.249 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[35809]: audit 2024-02-10T09:36:38.896057+0000 mon.a (mon.0) 760 : audit [INF] from='client.? 172.21.15.153:0/1598783978' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:36:40.249 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[35809]: cluster 2024-02-10T09:36:38.896104+0000 mon.a (mon.0) 761 : cluster [DBG] mgrmap e27: y(active, starting, since 0.360088s) 2024-02-10T09:36:40.249 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[35809]: audit 2024-02-10T09:36:39.543049+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.34643 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:36:40.249 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:39 smithi204 bash[35809]: cluster 2024-02-10T09:36:39.543093+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-10T09:36:40.249 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:40 smithi204 bash[44646]: debug 2024-02-10T09:36:40.246+0000 7fc29ec21680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:36:40.546 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:40 smithi204 bash[44646]: debug 2024-02-10T09:36:40.332+0000 7fc29ec21680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:36:40.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:40 smithi204 bash[44646]: debug 2024-02-10T09:36:40.545+0000 7fc29ec21680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:36:41.781 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:41 smithi204 bash[44646]: debug 2024-02-10T09:36:41.484+0000 7fc29ec21680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:36:41.781 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:41 smithi204 bash[44646]: debug 2024-02-10T09:36:41.612+0000 7fc29ec21680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:36:41.781 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:41 smithi204 bash[44646]: debug 2024-02-10T09:36:41.690+0000 7fc29ec21680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:36:42.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:41 smithi204 bash[44646]: debug 2024-02-10T09:36:41.780+0000 7fc29ec21680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:36:42.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:41 smithi204 bash[44646]: debug 2024-02-10T09:36:41.978+0000 7fc29ec21680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:36:42.184 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:41 smithi153 bash[61050]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:36:42.184 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 bash[61050]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:36:42.185 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 bash[61050]: Error: 3 errors occurred while pulling: 2024-02-10T09:36:42.185 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 bash[61050]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:36:42.185 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 bash[61050]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:36:42.185 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 bash[61050]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:36:42.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:42.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:42.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:42 smithi153 systemd[1]: Failed to start Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:42.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:42 smithi204 bash[44646]: debug 2024-02-10T09:36:42.256+0000 7fc29ec21680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:36:42.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:42 smithi204 bash[44646]: debug 2024-02-10T09:36:42.356+0000 7fc29ec21680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:36:43.352 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.086+0000 7fc29ec21680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:36:43.352 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.170+0000 7fc29ec21680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:36:43.353 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.264+0000 7fc29ec21680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:36:43.353 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[49655]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:36:43.353 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[49655]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:36:43.708 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.445+0000 7fc29ec21680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:36:43.709 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[49655]: Error: 3 errors occurred while pulling: 2024-02-10T09:36:43.709 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[49655]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:36:43.709 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[49655]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:36:43.709 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[49655]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:36:43.709 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Control process exited, code=exited status=125 2024-02-10T09:36:43.709 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:36:43.710 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:43 smithi204 systemd[1]: Failed to start Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:43.978 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.707+0000 7fc29ec21680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:36:43.978 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.841+0000 7fc29ec21680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:36:44.235 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:43 smithi204 bash[44646]: debug 2024-02-10T09:36:43.976+0000 7fc29ec21680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:36:44.236 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: debug 2024-02-10T09:36:44.149+0000 7fc29ec21680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:36:44.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: debug 2024-02-10T09:36:44.233+0000 7fc29ec21680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: [10/Feb/2024:09:36:44] ENGINE Bus STARTING 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: CherryPy Checker: 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: The Application mounted at '' has an empty config. 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: [10/Feb/2024:09:36:44] ENGINE Serving on http://:::9283 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[44646]: [10/Feb/2024:09:36:44] ENGINE Bus STARTED 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[35809]: cluster 2024-02-10T09:36:44.248916+0000 mon.a (mon.0) 764 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[35809]: audit 2024-02-10T09:36:44.249323+0000 mon.b (mon.2) 258 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[35809]: audit 2024-02-10T09:36:44.249772+0000 mon.b (mon.2) 259 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[35809]: audit 2024-02-10T09:36:44.251695+0000 mon.b (mon.2) 260 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:36:44.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:44 smithi204 bash[35809]: audit 2024-02-10T09:36:44.252548+0000 mon.b (mon.2) 261 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:44.615 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:36:44.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[39117]: cluster 2024-02-10T09:36:44.248916+0000 mon.a (mon.0) 764 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:36:44.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[39117]: audit 2024-02-10T09:36:44.249323+0000 mon.b (mon.2) 258 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:36:44.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[39117]: audit 2024-02-10T09:36:44.249772+0000 mon.b (mon.2) 259 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:44.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[39117]: audit 2024-02-10T09:36:44.251695+0000 mon.b (mon.2) 260 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:36:44.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[39117]: audit 2024-02-10T09:36:44.252548+0000 mon.b (mon.2) 261 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:44.616 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:36:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:36:44.252Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:36:44.616 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:36:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:36:44.615Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=2 2024-02-10T09:36:44.616 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[35780]: cluster 2024-02-10T09:36:44.248916+0000 mon.a (mon.0) 764 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:36:44.616 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[35780]: audit 2024-02-10T09:36:44.249323+0000 mon.b (mon.2) 258 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:36:44.616 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[35780]: audit 2024-02-10T09:36:44.249772+0000 mon.b (mon.2) 259 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:44.616 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[35780]: audit 2024-02-10T09:36:44.251695+0000 mon.b (mon.2) 260 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:36:44.616 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:44 smithi153 bash[35780]: audit 2024-02-10T09:36:44.252548+0000 mon.b (mon.2) 261 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:45.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:44] ENGINE Bus STOPPING 2024-02-10T09:36:45.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:36:45.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:44] ENGINE Bus STOPPED 2024-02-10T09:36:45.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:45] ENGINE Bus STARTING 2024-02-10T09:36:45.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [80B blob data] 2024-02-10T09:36:45.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [80B blob data] 2024-02-10T09:36:45.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: CherryPy Checker: 2024-02-10T09:36:45.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: The Application mounted at '' has an empty config. 2024-02-10T09:36:45.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:36:45.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:45] ENGINE Serving on http://:::9283 2024-02-10T09:36:45.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:36:45] ENGINE Bus STARTED 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: cluster 2024-02-10T09:36:44.307138+0000 mon.a (mon.0) 765 : cluster [DBG] mgrmap e28: y(active, starting, since 5s), standbys: x 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.307711+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.308004+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.897525+0000 mon.c (mon.1) 35 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.897693+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.897851+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.899516+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.899813+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.900029+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.900257+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.900495+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.900779+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:36:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.901060+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.901355+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.901628+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.901840+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.902301+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.903137+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: cluster 2024-02-10T09:36:44.307138+0000 mon.a (mon.0) 765 : cluster [DBG] mgrmap e28: y(active, starting, since 5s), standbys: x 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.307711+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.308004+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.897525+0000 mon.c (mon.1) 35 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:36:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.897693+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.897851+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.899516+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.899813+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.900029+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.900257+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.900495+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.900779+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.901060+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.901355+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.901628+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.901840+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.902301+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:36:45.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.903137+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.904111+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.904480+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.904838+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.905177+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.905560+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.905934+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.906259+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.906586+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:36:45.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.906948+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.907280+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.907581+0000 mon.c (mon.1) 60 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.907883+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.908199+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.908542+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.908868+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.909170+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:36:45.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.909460+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.904111+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.904480+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.904838+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.905177+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.905560+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.905934+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.906259+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.906586+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.906948+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.907280+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.907581+0000 mon.c (mon.1) 60 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.907883+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:36:45.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.908199+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.908542+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.908868+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.909170+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.909460+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.909754+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.910042+0000 mon.c (mon.1) 68 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.910344+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.910625+0000 mon.c (mon.1) 70 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.910920+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:36:45.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.911210+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.911500+0000 mon.c (mon.1) 73 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.911782+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: cluster 2024-02-10T09:36:44.939537+0000 mon.a (mon.0) 768 : cluster [INF] Manager daemon y is now available 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: cephadm 2024-02-10T09:36:44.952069+0000 mgr.y (mgr.14838) 1 : cephadm [INF] inventory: adjusted host smithi204 addr 'smithi204' -> '172.21.15.204' 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.952522+0000 mon.c (mon.1) 75 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.952884+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:44.956245+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.050531+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.052091+0000 mon.c (mon.1) 77 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.099113+0000 mon.c (mon.1) 78 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.099465+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.112680+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.113133+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.275221+0000 mon.c (mon.1) 80 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:36:45.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[35780]: audit 2024-02-10T09:36:45.276091+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.909754+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.910042+0000 mon.c (mon.1) 68 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.910344+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.910625+0000 mon.c (mon.1) 70 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.910920+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.911210+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:36:45.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.911500+0000 mon.c (mon.1) 73 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.911782+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: cluster 2024-02-10T09:36:44.939537+0000 mon.a (mon.0) 768 : cluster [INF] Manager daemon y is now available 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: cephadm 2024-02-10T09:36:44.952069+0000 mgr.y (mgr.14838) 1 : cephadm [INF] inventory: adjusted host smithi204 addr 'smithi204' -> '172.21.15.204' 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.952522+0000 mon.c (mon.1) 75 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.952884+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:44.956245+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.050531+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.052091+0000 mon.c (mon.1) 77 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.099113+0000 mon.c (mon.1) 78 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.099465+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:36:45.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.112680+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:36:45.450 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.113133+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:36:45.450 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.275221+0000 mon.c (mon.1) 80 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:36:45.450 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:45 smithi153 bash[39117]: audit 2024-02-10T09:36:45.276091+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: cluster 2024-02-10T09:36:44.307138+0000 mon.a (mon.0) 765 : cluster [DBG] mgrmap e28: y(active, starting, since 5s), standbys: x 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.307711+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.308004+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.? 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.897525+0000 mon.c (mon.1) 35 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.897693+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.897851+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.899516+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.899813+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.900029+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.900257+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.900495+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:36:45.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.900779+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.901060+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.901355+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.901628+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.901840+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.902301+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.903137+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:36:45.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:36:45.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:36:45.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:45.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:45.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[50951]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:45.584 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:45 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:45.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.904111+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:36:45.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.904480+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:36:45.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.904838+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:36:45.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.905177+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:36:45.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.905560+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:36:45.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.905934+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.906259+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.906586+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.906948+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.907280+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.907581+0000 mon.c (mon.1) 60 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.907883+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.908199+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.908542+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.908868+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.909170+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.909460+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.909754+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.910042+0000 mon.c (mon.1) 68 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.910344+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:36:45.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.910625+0000 mon.c (mon.1) 70 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.910920+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.911210+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.911500+0000 mon.c (mon.1) 73 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.911782+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: cluster 2024-02-10T09:36:44.939537+0000 mon.a (mon.0) 768 : cluster [INF] Manager daemon y is now available 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: cephadm 2024-02-10T09:36:44.952069+0000 mgr.y (mgr.14838) 1 : cephadm [INF] inventory: adjusted host smithi204 addr 'smithi204' -> '172.21.15.204' 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.952522+0000 mon.c (mon.1) 75 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.952884+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:44.956245+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/inventory","val":"{\"smithi153\": {\"hostname\": \"smithi153\", \"addr\": \"172.21.15.153\", \"labels\": [], \"status\": \"\"}, \"smithi204\": {\"hostname\": \"smithi204\", \"addr\": \"172.21.15.204\", \"labels\": [], \"status\": \"\"}}"}]': finished 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.050531+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.052091+0000 mon.c (mon.1) 77 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.099113+0000 mon.c (mon.1) 78 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.099465+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:36:45.587 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.112680+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:36:45.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.113133+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:36:45.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.275221+0000 mon.c (mon.1) 80 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:36:45.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:45 smithi204 bash[35809]: audit 2024-02-10T09:36:45.276091+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:36:46.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:46 smithi204 bash[35809]: cluster 2024-02-10T09:36:45.325700+0000 mon.a (mon.0) 773 : cluster [DBG] mgrmap e29: y(active, since 6s), standbys: x 2024-02-10T09:36:46.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:46 smithi204 bash[35809]: cluster 2024-02-10T09:36:45.340620+0000 mgr.y (mgr.14838) 2 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:46 smithi153 bash[39117]: cluster 2024-02-10T09:36:45.325700+0000 mon.a (mon.0) 773 : cluster [DBG] mgrmap e29: y(active, since 6s), standbys: x 2024-02-10T09:36:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:46 smithi153 bash[39117]: cluster 2024-02-10T09:36:45.340620+0000 mgr.y (mgr.14838) 2 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:46 smithi153 bash[35780]: cluster 2024-02-10T09:36:45.325700+0000 mon.a (mon.0) 773 : cluster [DBG] mgrmap e29: y(active, since 6s), standbys: x 2024-02-10T09:36:46.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:46 smithi153 bash[35780]: cluster 2024-02-10T09:36:45.340620+0000 mgr.y (mgr.14838) 2 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:47.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:47 smithi204 bash[35809]: cluster 2024-02-10T09:36:46.320444+0000 mon.a (mon.0) 774 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:36:47.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:47 smithi204 bash[35809]: cluster 2024-02-10T09:36:46.320475+0000 mon.a (mon.0) 775 : cluster [INF] Cluster is now healthy 2024-02-10T09:36:47.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:47 smithi204 bash[35809]: cluster 2024-02-10T09:36:46.901170+0000 mgr.y (mgr.14838) 3 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:47 smithi153 bash[35780]: cluster 2024-02-10T09:36:46.320444+0000 mon.a (mon.0) 774 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:36:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:47 smithi153 bash[35780]: cluster 2024-02-10T09:36:46.320475+0000 mon.a (mon.0) 775 : cluster [INF] Cluster is now healthy 2024-02-10T09:36:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:47 smithi153 bash[35780]: cluster 2024-02-10T09:36:46.901170+0000 mgr.y (mgr.14838) 3 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:47 smithi153 bash[39117]: cluster 2024-02-10T09:36:46.320444+0000 mon.a (mon.0) 774 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:36:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:47 smithi153 bash[39117]: cluster 2024-02-10T09:36:46.320475+0000 mon.a (mon.0) 775 : cluster [INF] Cluster is now healthy 2024-02-10T09:36:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:47 smithi153 bash[39117]: cluster 2024-02-10T09:36:46.901170+0000 mgr.y (mgr.14838) 3 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:49.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:48 smithi204 bash[35809]: cluster 2024-02-10T09:36:47.800931+0000 mon.a (mon.0) 776 : cluster [DBG] mgrmap e30: y(active, since 9s), standbys: x 2024-02-10T09:36:49.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:48 smithi153 bash[35780]: cluster 2024-02-10T09:36:47.800931+0000 mon.a (mon.0) 776 : cluster [DBG] mgrmap e30: y(active, since 9s), standbys: x 2024-02-10T09:36:49.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:48 smithi153 bash[39117]: cluster 2024-02-10T09:36:47.800931+0000 mon.a (mon.0) 776 : cluster [DBG] mgrmap e30: y(active, since 9s), standbys: x 2024-02-10T09:36:50.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: cluster 2024-02-10T09:36:48.901604+0000 mgr.y (mgr.14838) 4 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:50.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.051703+0000 mon.c (mon.1) 82 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.052196+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.055282+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.338188+0000 mon.c (mon.1) 83 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.338601+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.338931+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.339233+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.339518+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.339763+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.340013+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.340273+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:49 smithi204 bash[35809]: audit 2024-02-10T09:36:49.340558+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: cluster 2024-02-10T09:36:48.901604+0000 mgr.y (mgr.14838) 4 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:50.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: cluster 2024-02-10T09:36:48.901604+0000 mgr.y (mgr.14838) 4 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:50.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.051703+0000 mon.c (mon.1) 82 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.051703+0000 mon.c (mon.1) 82 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.052196+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.052196+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.055282+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.338188+0000 mon.c (mon.1) 83 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.338601+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.338931+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.339233+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.339518+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.339763+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.340013+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.093 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.340273+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[35780]: audit 2024-02-10T09:36:49.340558+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.055282+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.050398Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:04.530747Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.338188+0000 mon.c (mon.1) 83 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.338601+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.338931+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.339233+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.339518+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.339763+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.340013+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.340273+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:49 smithi153 bash[39117]: audit 2024-02-10T09:36:49.340558+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:49.842102+0000 mon.c (mon.1) 92 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:49.842547+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:49.846452+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.128903+0000 mon.c (mon.1) 93 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.129337+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.129683+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.130023+0000 mon.c (mon.1) 96 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.130342+0000 mon.c (mon.1) 97 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.130654+0000 mon.c (mon.1) 98 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.130954+0000 mon.c (mon.1) 99 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.131240+0000 mon.c (mon.1) 100 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.131520+0000 mon.c (mon.1) 101 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.132919+0000 mon.c (mon.1) 102 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.133414+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:50.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:50 smithi204 bash[35809]: audit 2024-02-10T09:36:50.136421+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:49.842102+0000 mon.c (mon.1) 92 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:49.842102+0000 mon.c (mon.1) 92 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:51.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:49.842547+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:51.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:49.846452+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:36:51.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.128903+0000 mon.c (mon.1) 93 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:49.842547+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:36:51.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:49.846452+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:36:49.841019Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:29:06.174259Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.128903+0000 mon.c (mon.1) 93 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.129337+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.129683+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.130023+0000 mon.c (mon.1) 96 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.130342+0000 mon.c (mon.1) 97 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.130654+0000 mon.c (mon.1) 98 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.130954+0000 mon.c (mon.1) 99 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.131240+0000 mon.c (mon.1) 100 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.131520+0000 mon.c (mon.1) 101 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.132919+0000 mon.c (mon.1) 102 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.133414+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[39117]: audit 2024-02-10T09:36:50.136421+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.129337+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.129683+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.130023+0000 mon.c (mon.1) 96 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.130342+0000 mon.c (mon.1) 97 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.130654+0000 mon.c (mon.1) 98 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.130954+0000 mon.c (mon.1) 99 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.131240+0000 mon.c (mon.1) 100 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:36:51.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.131520+0000 mon.c (mon.1) 101 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:36:51.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.132919+0000 mon.c (mon.1) 102 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:51.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.133414+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:51.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:50 smithi153 bash[35780]: audit 2024-02-10T09:36:50.136421+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:52.053 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:51 smithi204 bash[35809]: cluster 2024-02-10T09:36:50.132401+0000 mgr.y (mgr.14838) 5 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:52.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:51 smithi204 bash[35809]: cephadm 2024-02-10T09:36:50.143270+0000 mgr.y (mgr.14838) 6 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:52.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:51 smithi204 bash[35809]: cephadm 2024-02-10T09:36:50.179608+0000 mgr.y (mgr.14838) 7 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:52.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:51 smithi204 bash[35809]: cluster 2024-02-10T09:36:50.843839+0000 mon.a (mon.0) 783 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[35780]: cluster 2024-02-10T09:36:50.132401+0000 mgr.y (mgr.14838) 5 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[35780]: cephadm 2024-02-10T09:36:50.143270+0000 mgr.y (mgr.14838) 6 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[35780]: cephadm 2024-02-10T09:36:50.179608+0000 mgr.y (mgr.14838) 7 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[35780]: cluster 2024-02-10T09:36:50.843839+0000 mon.a (mon.0) 783 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[39117]: cluster 2024-02-10T09:36:50.132401+0000 mgr.y (mgr.14838) 5 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[39117]: cephadm 2024-02-10T09:36:50.143270+0000 mgr.y (mgr.14838) 6 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[39117]: cephadm 2024-02-10T09:36:50.179608+0000 mgr.y (mgr.14838) 7 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:52.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:51 smithi153 bash[39117]: cluster 2024-02-10T09:36:50.843839+0000 mon.a (mon.0) 783 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:36:52.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:52 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:36:52.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:52 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:36:52.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:52 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:52.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:52 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:52.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:52 smithi153 bash[64224]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:36:52.940 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:36:52 smithi153 bash[64224]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:36:53.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:53.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:53.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:53.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[51826]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:53.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:53.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:53.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:53.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:36:53.048+0000 7f10d82ee700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:53.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:53.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:36:53.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:36:53.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:53.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:53.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:53.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:53.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:36:53.769 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:36:53.769 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:36:53.769 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:36:53.770 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:53.770 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:53 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:53.770 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[51909]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:36:53.770 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[51909]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:36:54.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[35809]: cluster 2024-02-10T09:36:52.133251+0000 mgr.y (mgr.14838) 8 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[35809]: audit 2024-02-10T09:36:53.050794+0000 mon.c (mon.1) 103 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[35809]: audit 2024-02-10T09:36:53.053297+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[35809]: audit 2024-02-10T09:36:53.053939+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:54.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:53 smithi204 bash[35809]: audit 2024-02-10T09:36:53.059394+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[39117]: cluster 2024-02-10T09:36:52.133251+0000 mgr.y (mgr.14838) 8 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[39117]: audit 2024-02-10T09:36:53.050794+0000 mon.c (mon.1) 103 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[39117]: audit 2024-02-10T09:36:53.053297+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[39117]: audit 2024-02-10T09:36:53.053939+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[39117]: audit 2024-02-10T09:36:53.059394+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[35780]: cluster 2024-02-10T09:36:52.133251+0000 mgr.y (mgr.14838) 8 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[35780]: audit 2024-02-10T09:36:53.050794+0000 mon.c (mon.1) 103 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:36:54.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[35780]: audit 2024-02-10T09:36:53.053297+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:54.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[35780]: audit 2024-02-10T09:36:53.053939+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:36:54.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:53 smithi153 bash[35780]: audit 2024-02-10T09:36:53.059394+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:36:54.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:36:54] "GET /metrics HTTP/1.1" 200 32888 "" "Prometheus/2.33.4" 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: cephadm 2024-02-10T09:36:53.049122+0000 mgr.y (mgr.14838) 9 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: main() 2024-02-10T09:36:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:36:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: main() 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:55.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: cluster 2024-02-10T09:36:53.052515+0000 mgr.y (mgr.14838) 10 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: cluster 2024-02-10T09:36:53.052692+0000 mgr.y (mgr.14838) 11 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: cephadm 2024-02-10T09:36:53.065657+0000 mgr.y (mgr.14838) 12 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:55.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:54 smithi204 bash[35809]: cephadm 2024-02-10T09:36:53.072073+0000 mgr.y (mgr.14838) 13 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: cephadm 2024-02-10T09:36:53.049122+0000 mgr.y (mgr.14838) 9 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: main() 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:55.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: main() 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: cluster 2024-02-10T09:36:53.052515+0000 mgr.y (mgr.14838) 10 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: cluster 2024-02-10T09:36:53.052692+0000 mgr.y (mgr.14838) 11 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: cephadm 2024-02-10T09:36:53.065657+0000 mgr.y (mgr.14838) 12 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[39117]: cephadm 2024-02-10T09:36:53.072073+0000 mgr.y (mgr.14838) 13 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: cephadm 2024-02-10T09:36:53.049122+0000 mgr.y (mgr.14838) 9 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:36:55.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: main() 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:55.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:55.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: main() 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:55.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: cluster 2024-02-10T09:36:53.052515+0000 mgr.y (mgr.14838) 10 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:55.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: cluster 2024-02-10T09:36:53.052692+0000 mgr.y (mgr.14838) 11 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:55.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: cephadm 2024-02-10T09:36:53.065657+0000 mgr.y (mgr.14838) 12 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:36:55.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:54 smithi153 bash[35780]: cephadm 2024-02-10T09:36:53.072073+0000 mgr.y (mgr.14838) 13 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:36:56.081 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:36:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 bash[52041]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:36:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:36:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:36:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:36:55 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:36:55.944+0000 7f10d82ee700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:36:56.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:56.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:56.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:56.443 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:36:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:36:57.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: cluster 2024-02-10T09:36:55.053086+0000 mgr.y (mgr.14838) 14 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:57.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: cephadm 2024-02-10T09:36:55.945522+0000 mgr.y (mgr.14838) 15 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: main() 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:36:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:36:57.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: cluster 2024-02-10T09:36:55.053086+0000 mgr.y (mgr.14838) 14 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: cephadm 2024-02-10T09:36:55.945522+0000 mgr.y (mgr.14838) 15 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:57.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: main() 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: main() 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: main() 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:57.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:36:57.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:57.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:57.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:57.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:57.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:56 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: cluster 2024-02-10T09:36:55.053086+0000 mgr.y (mgr.14838) 14 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: cephadm 2024-02-10T09:36:55.945522+0000 mgr.y (mgr.14838) 15 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: main() 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: main() 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:36:57.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:36:57.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:56 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:36:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:36:58 smithi153 bash[35780]: cluster 2024-02-10T09:36:57.053813+0000 mgr.y (mgr.14838) 16 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:59.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:36:58 smithi153 bash[39117]: cluster 2024-02-10T09:36:57.053813+0000 mgr.y (mgr.14838) 16 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:36:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:36:58 smithi204 bash[35809]: cluster 2024-02-10T09:36:57.053813+0000 mgr.y (mgr.14838) 16 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:01.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:00 smithi153 bash[35780]: cluster 2024-02-10T09:36:59.054244+0000 mgr.y (mgr.14838) 17 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:01.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:00 smithi153 bash[39117]: cluster 2024-02-10T09:36:59.054244+0000 mgr.y (mgr.14838) 17 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:00 smithi204 bash[35809]: cluster 2024-02-10T09:36:59.054244+0000 mgr.y (mgr.14838) 17 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:03.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:02 smithi153 bash[35780]: cluster 2024-02-10T09:37:01.054948+0000 mgr.y (mgr.14838) 18 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:02 smithi153 bash[39117]: cluster 2024-02-10T09:37:01.054948+0000 mgr.y (mgr.14838) 18 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:03.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:02 smithi204 bash[35809]: cluster 2024-02-10T09:37:01.054948+0000 mgr.y (mgr.14838) 18 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:37:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:37:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:04] "GET /metrics HTTP/1.1" 200 153218 "" "Prometheus/2.33.4" 2024-02-10T09:37:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:04 smithi153 bash[39117]: cluster 2024-02-10T09:37:03.055429+0000 mgr.y (mgr.14838) 19 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:04 smithi153 bash[35780]: cluster 2024-02-10T09:37:03.055429+0000 mgr.y (mgr.14838) 19 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:04 smithi204 bash[35809]: cluster 2024-02-10T09:37:03.055429+0000 mgr.y (mgr.14838) 19 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:06.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 bash[52066]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:37:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:37:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:37:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:06 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:06 smithi153 bash[35780]: cluster 2024-02-10T09:37:05.055983+0000 mgr.y (mgr.14838) 20 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:06 smithi153 bash[39117]: cluster 2024-02-10T09:37:05.055983+0000 mgr.y (mgr.14838) 20 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:07.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:06 smithi204 bash[35809]: cluster 2024-02-10T09:37:05.055983+0000 mgr.y (mgr.14838) 20 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:08 smithi153 bash[39117]: cluster 2024-02-10T09:37:07.056916+0000 mgr.y (mgr.14838) 21 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:08 smithi153 bash[35780]: cluster 2024-02-10T09:37:07.056916+0000 mgr.y (mgr.14838) 21 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:09.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:08 smithi204 bash[35809]: cluster 2024-02-10T09:37:07.056916+0000 mgr.y (mgr.14838) 21 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:10 smithi153 bash[35780]: cluster 2024-02-10T09:37:09.057353+0000 mgr.y (mgr.14838) 22 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:11.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:10 smithi153 bash[39117]: cluster 2024-02-10T09:37:09.057353+0000 mgr.y (mgr.14838) 22 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:10 smithi204 bash[35809]: cluster 2024-02-10T09:37:09.057353+0000 mgr.y (mgr.14838) 22 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:13.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:12 smithi153 bash[35780]: cluster 2024-02-10T09:37:11.058104+0000 mgr.y (mgr.14838) 23 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:13.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:12 smithi153 bash[39117]: cluster 2024-02-10T09:37:11.058104+0000 mgr.y (mgr.14838) 23 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:13.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:12 smithi204 bash[35809]: cluster 2024-02-10T09:37:11.058104+0000 mgr.y (mgr.14838) 23 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:37:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:37:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:14] "GET /metrics HTTP/1.1" 200 153218 "" "Prometheus/2.33.4" 2024-02-10T09:37:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:14 smithi153 bash[35780]: cluster 2024-02-10T09:37:13.058694+0000 mgr.y (mgr.14838) 24 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:14 smithi153 bash[39117]: cluster 2024-02-10T09:37:13.058694+0000 mgr.y (mgr.14838) 24 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:14 smithi204 bash[35809]: cluster 2024-02-10T09:37:13.058694+0000 mgr.y (mgr.14838) 24 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:37:16.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 bash[52083]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:37:16.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:16 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:17.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:16 smithi153 bash[39117]: cluster 2024-02-10T09:37:15.059169+0000 mgr.y (mgr.14838) 25 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:17.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:16 smithi153 bash[35780]: cluster 2024-02-10T09:37:15.059169+0000 mgr.y (mgr.14838) 25 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:17.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:16 smithi204 bash[35809]: cluster 2024-02-10T09:37:15.059169+0000 mgr.y (mgr.14838) 25 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:19.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:18 smithi153 bash[35780]: cluster 2024-02-10T09:37:17.059816+0000 mgr.y (mgr.14838) 26 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:19.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:18 smithi153 bash[39117]: cluster 2024-02-10T09:37:17.059816+0000 mgr.y (mgr.14838) 26 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:19.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:18 smithi204 bash[35809]: cluster 2024-02-10T09:37:17.059816+0000 mgr.y (mgr.14838) 26 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:20 smithi153 bash[35780]: cluster 2024-02-10T09:37:19.060301+0000 mgr.y (mgr.14838) 27 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:21.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:20 smithi153 bash[39117]: cluster 2024-02-10T09:37:19.060301+0000 mgr.y (mgr.14838) 27 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:20 smithi204 bash[35809]: cluster 2024-02-10T09:37:19.060301+0000 mgr.y (mgr.14838) 27 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:23.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:22 smithi153 bash[39117]: cluster 2024-02-10T09:37:21.061030+0000 mgr.y (mgr.14838) 28 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:22 smithi153 bash[35780]: cluster 2024-02-10T09:37:21.061030+0000 mgr.y (mgr.14838) 28 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:22 smithi204 bash[35809]: cluster 2024-02-10T09:37:21.061030+0000 mgr.y (mgr.14838) 28 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:37:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:37:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:24] "GET /metrics HTTP/1.1" 200 153221 "" "Prometheus/2.33.4" 2024-02-10T09:37:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:24 smithi153 bash[35780]: cluster 2024-02-10T09:37:23.061563+0000 mgr.y (mgr.14838) 29 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:24 smithi153 bash[39117]: cluster 2024-02-10T09:37:23.061563+0000 mgr.y (mgr.14838) 29 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:25.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:24 smithi204 bash[35809]: cluster 2024-02-10T09:37:23.061563+0000 mgr.y (mgr.14838) 29 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:26.903 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 bash[52100]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:37:26.904 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:26 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:27.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:26 smithi153 bash[35780]: cluster 2024-02-10T09:37:25.062077+0000 mgr.y (mgr.14838) 30 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:26 smithi153 bash[39117]: cluster 2024-02-10T09:37:25.062077+0000 mgr.y (mgr.14838) 30 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:27.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:26 smithi204 bash[35809]: cluster 2024-02-10T09:37:25.062077+0000 mgr.y (mgr.14838) 30 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:29.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:28 smithi153 bash[35780]: cluster 2024-02-10T09:37:27.062796+0000 mgr.y (mgr.14838) 31 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:29.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:28 smithi153 bash[39117]: cluster 2024-02-10T09:37:27.062796+0000 mgr.y (mgr.14838) 31 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:29.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:37:29 smithi153 bash[64224]: time="2024-02-10T09:37:29Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:37:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:28 smithi204 bash[35809]: cluster 2024-02-10T09:37:27.062796+0000 mgr.y (mgr.14838) 31 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:30.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:37:30 smithi204 bash[51909]: time="2024-02-10T09:37:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:37:31.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:30 smithi153 bash[35780]: cluster 2024-02-10T09:37:29.063277+0000 mgr.y (mgr.14838) 32 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:31.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:30 smithi153 bash[39117]: cluster 2024-02-10T09:37:29.063277+0000 mgr.y (mgr.14838) 32 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:30 smithi204 bash[35809]: cluster 2024-02-10T09:37:29.063277+0000 mgr.y (mgr.14838) 32 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:33.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:32 smithi153 bash[35780]: cluster 2024-02-10T09:37:31.063994+0000 mgr.y (mgr.14838) 33 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:33.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:32 smithi153 bash[39117]: cluster 2024-02-10T09:37:31.063994+0000 mgr.y (mgr.14838) 33 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:33.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:32 smithi204 bash[35809]: cluster 2024-02-10T09:37:31.063994+0000 mgr.y (mgr.14838) 33 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:37:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:37:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:34] "GET /metrics HTTP/1.1" 200 153224 "" "Prometheus/2.33.4" 2024-02-10T09:37:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:34 smithi153 bash[35780]: cluster 2024-02-10T09:37:33.064578+0000 mgr.y (mgr.14838) 34 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:34 smithi153 bash[39117]: cluster 2024-02-10T09:37:33.064578+0000 mgr.y (mgr.14838) 34 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:34 smithi204 bash[35809]: cluster 2024-02-10T09:37:33.064578+0000 mgr.y (mgr.14838) 34 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:36 smithi153 bash[39117]: cluster 2024-02-10T09:37:35.065116+0000 mgr.y (mgr.14838) 35 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:36 smithi153 bash[35780]: cluster 2024-02-10T09:37:35.065116+0000 mgr.y (mgr.14838) 35 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:36 smithi204 bash[35809]: cluster 2024-02-10T09:37:35.065116+0000 mgr.y (mgr.14838) 35 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:37.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:37.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:37.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:37.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 bash[52118]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:37:37.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:37:37.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:37:37.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:36 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:39.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:38 smithi153 bash[35780]: cluster 2024-02-10T09:37:37.065854+0000 mgr.y (mgr.14838) 36 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:38 smithi153 bash[39117]: cluster 2024-02-10T09:37:37.065854+0000 mgr.y (mgr.14838) 36 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:39.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:38 smithi204 bash[35809]: cluster 2024-02-10T09:37:37.065854+0000 mgr.y (mgr.14838) 36 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:41.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:40 smithi204 bash[35809]: cluster 2024-02-10T09:37:39.066305+0000 mgr.y (mgr.14838) 37 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:40 smithi153 bash[35780]: cluster 2024-02-10T09:37:39.066305+0000 mgr.y (mgr.14838) 37 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:41.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:40 smithi153 bash[39117]: cluster 2024-02-10T09:37:39.066305+0000 mgr.y (mgr.14838) 37 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:42 smithi204 bash[35809]: cluster 2024-02-10T09:37:41.067214+0000 mgr.y (mgr.14838) 38 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:43.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:42 smithi153 bash[39117]: cluster 2024-02-10T09:37:41.067214+0000 mgr.y (mgr.14838) 38 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:43.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:42 smithi153 bash[35780]: cluster 2024-02-10T09:37:41.067214+0000 mgr.y (mgr.14838) 38 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:37:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:37:44.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:44] "GET /metrics HTTP/1.1" 200 153224 "" "Prometheus/2.33.4" 2024-02-10T09:37:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: cluster 2024-02-10T09:37:43.067595+0000 mgr.y (mgr.14838) 39 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.945684+0000 mon.c (mon.1) 105 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:37:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.945890+0000 mon.c (mon.1) 106 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.946022+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:37:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.946097+0000 mon.c (mon.1) 107 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.946324+0000 mon.c (mon.1) 108 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:37:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.946483+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.946773+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:44 smithi204 bash[35809]: audit 2024-02-10T09:37:44.946984+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: cluster 2024-02-10T09:37:43.067595+0000 mgr.y (mgr.14838) 39 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.945684+0000 mon.c (mon.1) 105 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.945890+0000 mon.c (mon.1) 106 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.946022+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.946097+0000 mon.c (mon.1) 107 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.946324+0000 mon.c (mon.1) 108 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.946483+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.946773+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[39117]: audit 2024-02-10T09:37:44.946984+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: cluster 2024-02-10T09:37:43.067595+0000 mgr.y (mgr.14838) 39 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.945684+0000 mon.c (mon.1) 105 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.945890+0000 mon.c (mon.1) 106 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.946022+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.946097+0000 mon.c (mon.1) 107 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.946324+0000 mon.c (mon.1) 108 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.946483+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.946773+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:37:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:44 smithi153 bash[35780]: audit 2024-02-10T09:37:44.946984+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-10T09:37:47.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:46 smithi204 bash[35809]: cluster 2024-02-10T09:37:45.067999+0000 mgr.y (mgr.14838) 40 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:47.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:47 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:37:47.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:47 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:37:47.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:47 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:47.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:47 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-02-10T09:37:47.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:47 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:37:47.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:47 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:47.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:46 smithi153 bash[35780]: cluster 2024-02-10T09:37:45.067999+0000 mgr.y (mgr.14838) 40 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:47.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:46 smithi153 bash[39117]: cluster 2024-02-10T09:37:45.067999+0000 mgr.y (mgr.14838) 40 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:49.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:48 smithi204 bash[35809]: cluster 2024-02-10T09:37:47.068775+0000 mgr.y (mgr.14838) 41 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:49.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:48 smithi153 bash[35780]: cluster 2024-02-10T09:37:47.068775+0000 mgr.y (mgr.14838) 41 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:49.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:48 smithi153 bash[39117]: cluster 2024-02-10T09:37:47.068775+0000 mgr.y (mgr.14838) 41 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:51.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:50 smithi204 bash[35809]: cluster 2024-02-10T09:37:49.069194+0000 mgr.y (mgr.14838) 42 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:51.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:50 smithi153 bash[35780]: cluster 2024-02-10T09:37:49.069194+0000 mgr.y (mgr.14838) 42 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:51.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:50 smithi153 bash[39117]: cluster 2024-02-10T09:37:49.069194+0000 mgr.y (mgr.14838) 42 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:52 smithi204 bash[35809]: cluster 2024-02-10T09:37:51.069872+0000 mgr.y (mgr.14838) 43 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:52 smithi153 bash[35780]: cluster 2024-02-10T09:37:51.069872+0000 mgr.y (mgr.14838) 43 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:53.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:52 smithi153 bash[39117]: cluster 2024-02-10T09:37:51.069872+0000 mgr.y (mgr.14838) 43 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:37:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:37:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:37:54] "GET /metrics HTTP/1.1" 200 153216 "" "Prometheus/2.33.4" 2024-02-10T09:37:55.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:54 smithi204 bash[35809]: cluster 2024-02-10T09:37:53.070434+0000 mgr.y (mgr.14838) 44 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:54 smithi153 bash[35780]: cluster 2024-02-10T09:37:53.070434+0000 mgr.y (mgr.14838) 44 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:55.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:54 smithi153 bash[39117]: cluster 2024-02-10T09:37:53.070434+0000 mgr.y (mgr.14838) 44 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:55 smithi204 bash[35809]: audit 2024-02-10T09:37:55.947109+0000 mon.c (mon.1) 109 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:37:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:55 smithi153 bash[35780]: audit 2024-02-10T09:37:55.947109+0000 mon.c (mon.1) 109 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:37:56.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:55 smithi153 bash[39117]: audit 2024-02-10T09:37:55.947109+0000 mon.c (mon.1) 109 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:37:57.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:56 smithi204 bash[35809]: cluster 2024-02-10T09:37:55.070941+0000 mgr.y (mgr.14838) 45 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:57.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:56 smithi204 bash[35809]: audit 2024-02-10T09:37:56.248487+0000 mon.c (mon.1) 110 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:37:57.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:56 smithi204 bash[35809]: audit 2024-02-10T09:37:56.249387+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:37:57.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:56 smithi204 bash[35809]: audit 2024-02-10T09:37:56.254506+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:37:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[39117]: cluster 2024-02-10T09:37:55.070941+0000 mgr.y (mgr.14838) 45 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[39117]: audit 2024-02-10T09:37:56.248487+0000 mon.c (mon.1) 110 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:37:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[39117]: audit 2024-02-10T09:37:56.249387+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:37:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[39117]: audit 2024-02-10T09:37:56.254506+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:37:57.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[35780]: cluster 2024-02-10T09:37:55.070941+0000 mgr.y (mgr.14838) 45 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:57.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[35780]: audit 2024-02-10T09:37:56.248487+0000 mon.c (mon.1) 110 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:37:57.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[35780]: audit 2024-02-10T09:37:56.249387+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:37:57.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:56 smithi153 bash[35780]: audit 2024-02-10T09:37:56.254506+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:37:58.127 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:58 smithi204 bash[35809]: cluster 2024-02-10T09:37:56.247848+0000 mgr.y (mgr.14838) 46 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:58.127 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:58 smithi204 bash[35809]: cluster 2024-02-10T09:37:56.248025+0000 mgr.y (mgr.14838) 47 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:58.127 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:58 smithi204 bash[35809]: cephadm 2024-02-10T09:37:56.258228+0000 mgr.y (mgr.14838) 48 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:37:58.127 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:37:58 smithi204 bash[35809]: cephadm 2024-02-10T09:37:56.264877+0000 mgr.y (mgr.14838) 49 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[39117]: cluster 2024-02-10T09:37:56.247848+0000 mgr.y (mgr.14838) 46 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[39117]: cluster 2024-02-10T09:37:56.248025+0000 mgr.y (mgr.14838) 47 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[39117]: cephadm 2024-02-10T09:37:56.258228+0000 mgr.y (mgr.14838) 48 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[39117]: cephadm 2024-02-10T09:37:56.264877+0000 mgr.y (mgr.14838) 49 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[35780]: cluster 2024-02-10T09:37:56.247848+0000 mgr.y (mgr.14838) 46 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[35780]: cluster 2024-02-10T09:37:56.248025+0000 mgr.y (mgr.14838) 47 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:37:58.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[35780]: cephadm 2024-02-10T09:37:56.258228+0000 mgr.y (mgr.14838) 48 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:37:58.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:37:58 smithi153 bash[35780]: cephadm 2024-02-10T09:37:56.264877+0000 mgr.y (mgr.14838) 49 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:37:59.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:37:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:37:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 bash[52347]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:37:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:37:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:37:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:37:59 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:37:59.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:37:59.162+0000 7f10d82ee700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:37:59.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:37:59.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:37:59.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:37:59.441 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:37:59.442 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:37:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:38:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:00 smithi204 bash[35809]: cluster 2024-02-10T09:37:58.248509+0000 mgr.y (mgr.14838) 50 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:00 smithi153 bash[35780]: cluster 2024-02-10T09:37:58.248509+0000 mgr.y (mgr.14838) 50 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:00 smithi153 bash[39117]: cluster 2024-02-10T09:37:58.248509+0000 mgr.y (mgr.14838) 50 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: cephadm 2024-02-10T09:37:59.163546+0000 mgr.y (mgr.14838) 51 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: main() 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:38:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:38:01.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: main() 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:38:01.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:01 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: cephadm 2024-02-10T09:37:59.163546+0000 mgr.y (mgr.14838) 51 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:38:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:38:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: main() 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:38:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:38:01.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: cephadm 2024-02-10T09:37:59.163546+0000 mgr.y (mgr.14838) 51 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:38:01.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: main() 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:38:01.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:38:01.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: main() 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:38:01.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:38:01.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:38:01.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:38:01.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:01.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:38:01.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:38:01.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: main() 2024-02-10T09:38:01.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:38:01.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:38:01.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:01 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:38:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:02 smithi204 bash[35809]: cluster 2024-02-10T09:38:00.249000+0000 mgr.y (mgr.14838) 52 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:02.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:02 smithi153 bash[35780]: cluster 2024-02-10T09:38:00.249000+0000 mgr.y (mgr.14838) 52 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:02.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:02 smithi153 bash[39117]: cluster 2024-02-10T09:38:00.249000+0000 mgr.y (mgr.14838) 52 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:04.021 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:38:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:38:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:04 smithi204 bash[35809]: cluster 2024-02-10T09:38:02.249797+0000 mgr.y (mgr.14838) 53 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:04 smithi153 bash[39117]: cluster 2024-02-10T09:38:02.249797+0000 mgr.y (mgr.14838) 53 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:04 smithi153 bash[35780]: cluster 2024-02-10T09:38:02.249797+0000 mgr.y (mgr.14838) 53 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:38:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:04] "GET /metrics HTTP/1.1" 200 153227 "" "Prometheus/2.33.4" 2024-02-10T09:38:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:06 smithi204 bash[35809]: cluster 2024-02-10T09:38:04.250164+0000 mgr.y (mgr.14838) 54 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:06.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:06 smithi153 bash[35780]: cluster 2024-02-10T09:38:04.250164+0000 mgr.y (mgr.14838) 54 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:06.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:06 smithi153 bash[39117]: cluster 2024-02-10T09:38:04.250164+0000 mgr.y (mgr.14838) 54 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:06.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:38:06 smithi153 bash[64224]: time="2024-02-10T09:38:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:38:07.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:38:07 smithi204 bash[51909]: time="2024-02-10T09:38:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:38:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:08 smithi204 bash[35809]: cluster 2024-02-10T09:38:06.251124+0000 mgr.y (mgr.14838) 55 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:08 smithi153 bash[35780]: cluster 2024-02-10T09:38:06.251124+0000 mgr.y (mgr.14838) 55 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:08 smithi153 bash[39117]: cluster 2024-02-10T09:38:06.251124+0000 mgr.y (mgr.14838) 55 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:09.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:38:09.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:38:09.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:09.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:09.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:09.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 bash[52373]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:38:09.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:38:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:09 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:10 smithi204 bash[35809]: cluster 2024-02-10T09:38:08.251554+0000 mgr.y (mgr.14838) 56 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:10 smithi153 bash[35780]: cluster 2024-02-10T09:38:08.251554+0000 mgr.y (mgr.14838) 56 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:10.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:10 smithi153 bash[39117]: cluster 2024-02-10T09:38:08.251554+0000 mgr.y (mgr.14838) 56 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:12 smithi204 bash[35809]: cluster 2024-02-10T09:38:10.251988+0000 mgr.y (mgr.14838) 57 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:12 smithi153 bash[35780]: cluster 2024-02-10T09:38:10.251988+0000 mgr.y (mgr.14838) 57 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:12.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:12 smithi153 bash[39117]: cluster 2024-02-10T09:38:10.251988+0000 mgr.y (mgr.14838) 57 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:14.052 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:38:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:38:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:14 smithi204 bash[35809]: cluster 2024-02-10T09:38:12.252661+0000 mgr.y (mgr.14838) 58 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:14.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:14 smithi153 bash[39117]: cluster 2024-02-10T09:38:12.252661+0000 mgr.y (mgr.14838) 58 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:14.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:14 smithi153 bash[35780]: cluster 2024-02-10T09:38:12.252661+0000 mgr.y (mgr.14838) 58 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:38:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:14] "GET /metrics HTTP/1.1" 200 153227 "" "Prometheus/2.33.4" 2024-02-10T09:38:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:16 smithi204 bash[35809]: cluster 2024-02-10T09:38:14.253105+0000 mgr.y (mgr.14838) 59 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:16.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:16 smithi153 bash[35780]: cluster 2024-02-10T09:38:14.253105+0000 mgr.y (mgr.14838) 59 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:16.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:16 smithi153 bash[39117]: cluster 2024-02-10T09:38:14.253105+0000 mgr.y (mgr.14838) 59 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:18.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:17 smithi204 bash[35809]: cluster 2024-02-10T09:38:16.253812+0000 mgr.y (mgr.14838) 60 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:17 smithi153 bash[35780]: cluster 2024-02-10T09:38:16.253812+0000 mgr.y (mgr.14838) 60 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:17 smithi153 bash[39117]: cluster 2024-02-10T09:38:16.253812+0000 mgr.y (mgr.14838) 60 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:19.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 bash[52391]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:38:19.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:19 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:19 smithi153 bash[35780]: cluster 2024-02-10T09:38:18.254380+0000 mgr.y (mgr.14838) 61 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:19 smithi153 bash[39117]: cluster 2024-02-10T09:38:18.254380+0000 mgr.y (mgr.14838) 61 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:19 smithi204 bash[35809]: cluster 2024-02-10T09:38:18.254380+0000 mgr.y (mgr.14838) 61 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:22.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:21 smithi153 bash[35780]: cluster 2024-02-10T09:38:20.254841+0000 mgr.y (mgr.14838) 62 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:22.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:21 smithi153 bash[39117]: cluster 2024-02-10T09:38:20.254841+0000 mgr.y (mgr.14838) 62 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:22.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:21 smithi204 bash[35809]: cluster 2024-02-10T09:38:20.254841+0000 mgr.y (mgr.14838) 62 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:24.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:23 smithi204 bash[35809]: cluster 2024-02-10T09:38:22.255489+0000 mgr.y (mgr.14838) 63 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:24.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:38:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:38:24.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:23 smithi153 bash[35780]: cluster 2024-02-10T09:38:22.255489+0000 mgr.y (mgr.14838) 63 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:24.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:23 smithi153 bash[39117]: cluster 2024-02-10T09:38:22.255489+0000 mgr.y (mgr.14838) 63 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:38:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:24] "GET /metrics HTTP/1.1" 200 153219 "" "Prometheus/2.33.4" 2024-02-10T09:38:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:25 smithi153 bash[35780]: cluster 2024-02-10T09:38:24.255958+0000 mgr.y (mgr.14838) 64 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:25 smithi153 bash[39117]: cluster 2024-02-10T09:38:24.255958+0000 mgr.y (mgr.14838) 64 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:25 smithi204 bash[35809]: cluster 2024-02-10T09:38:24.255958+0000 mgr.y (mgr.14838) 64 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:27 smithi153 bash[35780]: cluster 2024-02-10T09:38:26.256891+0000 mgr.y (mgr.14838) 65 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:27 smithi153 bash[39117]: cluster 2024-02-10T09:38:26.256891+0000 mgr.y (mgr.14838) 65 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:27 smithi204 bash[35809]: cluster 2024-02-10T09:38:26.256891+0000 mgr.y (mgr.14838) 65 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:29 smithi153 bash[35780]: cluster 2024-02-10T09:38:28.257444+0000 mgr.y (mgr.14838) 66 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:29 smithi153 bash[39117]: cluster 2024-02-10T09:38:28.257444+0000 mgr.y (mgr.14838) 66 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:29 smithi204 bash[35809]: cluster 2024-02-10T09:38:28.257444+0000 mgr.y (mgr.14838) 66 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:30.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:30.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 bash[52409]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:38:30.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:29 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:31 smithi153 bash[39117]: cluster 2024-02-10T09:38:30.258061+0000 mgr.y (mgr.14838) 67 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:31 smithi153 bash[35780]: cluster 2024-02-10T09:38:30.258061+0000 mgr.y (mgr.14838) 67 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:32.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:31 smithi204 bash[35809]: cluster 2024-02-10T09:38:30.258061+0000 mgr.y (mgr.14838) 67 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:33.870 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:38:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:38:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:33 smithi153 bash[35780]: cluster 2024-02-10T09:38:32.259014+0000 mgr.y (mgr.14838) 68 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:33 smithi153 bash[39117]: cluster 2024-02-10T09:38:32.259014+0000 mgr.y (mgr.14838) 68 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:33 smithi204 bash[35809]: cluster 2024-02-10T09:38:32.259014+0000 mgr.y (mgr.14838) 68 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:38:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:34] "GET /metrics HTTP/1.1" 200 153221 "" "Prometheus/2.33.4" 2024-02-10T09:38:36.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:35 smithi153 bash[35780]: cluster 2024-02-10T09:38:34.259445+0000 mgr.y (mgr.14838) 69 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:36.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:35 smithi153 bash[39117]: cluster 2024-02-10T09:38:34.259445+0000 mgr.y (mgr.14838) 69 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:35 smithi204 bash[35809]: cluster 2024-02-10T09:38:34.259445+0000 mgr.y (mgr.14838) 69 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:37 smithi153 bash[39117]: cluster 2024-02-10T09:38:36.260190+0000 mgr.y (mgr.14838) 70 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:37 smithi153 bash[35780]: cluster 2024-02-10T09:38:36.260190+0000 mgr.y (mgr.14838) 70 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:37 smithi204 bash[35809]: cluster 2024-02-10T09:38:36.260190+0000 mgr.y (mgr.14838) 70 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:40.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:39 smithi153 bash[35780]: cluster 2024-02-10T09:38:38.260670+0000 mgr.y (mgr.14838) 71 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:39 smithi153 bash[39117]: cluster 2024-02-10T09:38:38.260670+0000 mgr.y (mgr.14838) 71 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:40.197 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:39 smithi204 bash[35809]: cluster 2024-02-10T09:38:38.260670+0000 mgr.y (mgr.14838) 71 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:38:40.198 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:38:40.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:40.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:38:40.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:40.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:38:40.199 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:38:40.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 bash[52426]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:38:40.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:38:40.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:38:40.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:40 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:42.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:41 smithi153 bash[35780]: cluster 2024-02-10T09:38:40.261160+0000 mgr.y (mgr.14838) 72 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:42.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:41 smithi153 bash[39117]: cluster 2024-02-10T09:38:40.261160+0000 mgr.y (mgr.14838) 72 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:42.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:41 smithi204 bash[35809]: cluster 2024-02-10T09:38:40.261160+0000 mgr.y (mgr.14838) 72 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:43.897 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:38:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:38:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:43 smithi153 bash[35780]: cluster 2024-02-10T09:38:42.261862+0000 mgr.y (mgr.14838) 73 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:43 smithi153 bash[39117]: cluster 2024-02-10T09:38:42.261862+0000 mgr.y (mgr.14838) 73 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:44.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:38:43 smithi153 bash[64224]: time="2024-02-10T09:38:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:38:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:43 smithi204 bash[35809]: cluster 2024-02-10T09:38:42.261862+0000 mgr.y (mgr.14838) 73 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:38:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:44] "GET /metrics HTTP/1.1" 200 153221 "" "Prometheus/2.33.4" 2024-02-10T09:38:45.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:38:44 smithi204 bash[51909]: time="2024-02-10T09:38:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:38:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: cluster 2024-02-10T09:38:44.262357+0000 mgr.y (mgr.14838) 74 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.949721+0000 mon.c (mon.1) 111 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-02-10T09:38:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.950011+0000 mon.c (mon.1) 112 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.950158+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-02-10T09:38:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.950304+0000 mon.c (mon.1) 113 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:38:46.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.950605+0000 mon.c (mon.1) 114 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.950838+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.950887+0000 mon.c (mon.1) 115 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.951186+0000 mon.c (mon.1) 116 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.951248+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.951750+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.952029+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[35780]: audit 2024-02-10T09:38:44.952272+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: cluster 2024-02-10T09:38:44.262357+0000 mgr.y (mgr.14838) 74 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.949721+0000 mon.c (mon.1) 111 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.950011+0000 mon.c (mon.1) 112 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.950158+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.950304+0000 mon.c (mon.1) 113 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:38:46.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.950605+0000 mon.c (mon.1) 114 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.950838+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.950887+0000 mon.c (mon.1) 115 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.951186+0000 mon.c (mon.1) 116 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.951248+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.951750+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.952029+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-02-10T09:38:46.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:45 smithi153 bash[39117]: audit 2024-02-10T09:38:44.952272+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: cluster 2024-02-10T09:38:44.262357+0000 mgr.y (mgr.14838) 74 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.949721+0000 mon.c (mon.1) 111 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.950011+0000 mon.c (mon.1) 112 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.950158+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.950304+0000 mon.c (mon.1) 113 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.950605+0000 mon.c (mon.1) 114 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.950838+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.950887+0000 mon.c (mon.1) 115 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.951186+0000 mon.c (mon.1) 116 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.951248+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.951750+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-10T09:38:46.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.952029+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-02-10T09:38:46.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:45 smithi204 bash[35809]: audit 2024-02-10T09:38:44.952272+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:38:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:47 smithi153 bash[35780]: cluster 2024-02-10T09:38:46.263264+0000 mgr.y (mgr.14838) 75 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:47 smithi153 bash[39117]: cluster 2024-02-10T09:38:46.263264+0000 mgr.y (mgr.14838) 75 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:47 smithi204 bash[35809]: cluster 2024-02-10T09:38:46.263264+0000 mgr.y (mgr.14838) 75 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:49 smithi153 bash[35780]: cluster 2024-02-10T09:38:48.263850+0000 mgr.y (mgr.14838) 76 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:50.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:49 smithi153 bash[39117]: cluster 2024-02-10T09:38:48.263850+0000 mgr.y (mgr.14838) 76 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:49 smithi204 bash[35809]: cluster 2024-02-10T09:38:48.263850+0000 mgr.y (mgr.14838) 76 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:50.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:50 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:38:50.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:50 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:38:50.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:50 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:50.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:50 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-02-10T09:38:50.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:50 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:38:50.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:38:50 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:38:52.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:51 smithi153 bash[35780]: cluster 2024-02-10T09:38:50.264389+0000 mgr.y (mgr.14838) 77 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:51 smithi153 bash[39117]: cluster 2024-02-10T09:38:50.264389+0000 mgr.y (mgr.14838) 77 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:52.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:51 smithi204 bash[35809]: cluster 2024-02-10T09:38:50.264389+0000 mgr.y (mgr.14838) 77 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:53.927 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:38:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:38:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:53 smithi153 bash[39117]: cluster 2024-02-10T09:38:52.265287+0000 mgr.y (mgr.14838) 78 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:53 smithi153 bash[35780]: cluster 2024-02-10T09:38:52.265287+0000 mgr.y (mgr.14838) 78 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:53 smithi204 bash[35809]: cluster 2024-02-10T09:38:52.265287+0000 mgr.y (mgr.14838) 78 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:38:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:38:54] "GET /metrics HTTP/1.1" 200 153220 "" "Prometheus/2.33.4" 2024-02-10T09:38:56.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:55 smithi153 bash[35780]: cluster 2024-02-10T09:38:54.265820+0000 mgr.y (mgr.14838) 79 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:55 smithi153 bash[39117]: cluster 2024-02-10T09:38:54.265820+0000 mgr.y (mgr.14838) 79 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:55 smithi204 bash[35809]: cluster 2024-02-10T09:38:54.265820+0000 mgr.y (mgr.14838) 79 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:57 smithi153 bash[35780]: cluster 2024-02-10T09:38:56.266469+0000 mgr.y (mgr.14838) 80 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:57 smithi153 bash[39117]: cluster 2024-02-10T09:38:56.266469+0000 mgr.y (mgr.14838) 80 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:38:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:57 smithi204 bash[35809]: cluster 2024-02-10T09:38:56.266469+0000 mgr.y (mgr.14838) 80 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:59 smithi204 bash[35809]: cluster 2024-02-10T09:38:58.267043+0000 mgr.y (mgr.14838) 81 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:59 smithi204 bash[35809]: audit 2024-02-10T09:38:59.165112+0000 mon.c (mon.1) 117 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:39:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:59 smithi204 bash[35809]: audit 2024-02-10T09:38:59.466012+0000 mon.c (mon.1) 118 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:39:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:59 smithi204 bash[35809]: audit 2024-02-10T09:38:59.466885+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:39:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:38:59 smithi204 bash[35809]: audit 2024-02-10T09:38:59.470203+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:39:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[35780]: cluster 2024-02-10T09:38:58.267043+0000 mgr.y (mgr.14838) 81 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[35780]: audit 2024-02-10T09:38:59.165112+0000 mon.c (mon.1) 117 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:39:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[35780]: audit 2024-02-10T09:38:59.466012+0000 mon.c (mon.1) 118 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:39:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[35780]: audit 2024-02-10T09:38:59.466885+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:39:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[35780]: audit 2024-02-10T09:38:59.470203+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:39:00.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[39117]: cluster 2024-02-10T09:38:58.267043+0000 mgr.y (mgr.14838) 81 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[39117]: audit 2024-02-10T09:38:59.165112+0000 mon.c (mon.1) 117 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:39:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[39117]: audit 2024-02-10T09:38:59.466012+0000 mon.c (mon.1) 118 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:39:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[39117]: audit 2024-02-10T09:38:59.466885+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:39:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:38:59 smithi153 bash[39117]: audit 2024-02-10T09:38:59.470203+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:39:01.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:00 smithi204 bash[35809]: cluster 2024-02-10T09:38:59.465392+0000 mgr.y (mgr.14838) 82 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:01.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:00 smithi204 bash[35809]: cluster 2024-02-10T09:38:59.465539+0000 mgr.y (mgr.14838) 83 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:01.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:00 smithi204 bash[35809]: cephadm 2024-02-10T09:38:59.473928+0000 mgr.y (mgr.14838) 84 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:39:01.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:00 smithi204 bash[35809]: cephadm 2024-02-10T09:38:59.480104+0000 mgr.y (mgr.14838) 85 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:39:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[35780]: cluster 2024-02-10T09:38:59.465392+0000 mgr.y (mgr.14838) 82 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[35780]: cluster 2024-02-10T09:38:59.465539+0000 mgr.y (mgr.14838) 83 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[35780]: cephadm 2024-02-10T09:38:59.473928+0000 mgr.y (mgr.14838) 84 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:39:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[35780]: cephadm 2024-02-10T09:38:59.480104+0000 mgr.y (mgr.14838) 85 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:39:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[39117]: cluster 2024-02-10T09:38:59.465392+0000 mgr.y (mgr.14838) 82 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[39117]: cluster 2024-02-10T09:38:59.465539+0000 mgr.y (mgr.14838) 83 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[39117]: cephadm 2024-02-10T09:38:59.473928+0000 mgr.y (mgr.14838) 84 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:39:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:00 smithi153 bash[39117]: cephadm 2024-02-10T09:38:59.480104+0000 mgr.y (mgr.14838) 85 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:39:02.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:02.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 bash[52659]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:39:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:39:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:02.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:02 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:02.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:02.248+0000 7f10d82ee700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:02.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:02.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:02.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:02.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:02.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:02.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:39:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:02 smithi204 bash[35809]: cluster 2024-02-10T09:39:01.466209+0000 mgr.y (mgr.14838) 86 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:03.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:02 smithi153 bash[35780]: cluster 2024-02-10T09:39:01.466209+0000 mgr.y (mgr.14838) 86 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:03.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:02 smithi153 bash[39117]: cluster 2024-02-10T09:39:01.466209+0000 mgr.y (mgr.14838) 86 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:03.959 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: cephadm 2024-02-10T09:39:02.248806+0000 mgr.y (mgr.14838) 87 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: main() 2024-02-10T09:39:04.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:39:04.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: main() 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:39:04.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:03 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: cephadm 2024-02-10T09:39:02.248806+0000 mgr.y (mgr.14838) 87 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:39:04.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:04.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: main() 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:39:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:39:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: cephadm 2024-02-10T09:39:02.248806+0000 mgr.y (mgr.14838) 87 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: main() 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:04.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:39:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:04.420 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.421 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: main() 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: main() 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:39:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:39:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:03 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:39:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:04] "GET /metrics HTTP/1.1" 200 153229 "" "Prometheus/2.33.4" 2024-02-10T09:39:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:04 smithi204 bash[35809]: cluster 2024-02-10T09:39:03.466751+0000 mgr.y (mgr.14838) 88 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:05.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:04 smithi153 bash[39117]: cluster 2024-02-10T09:39:03.466751+0000 mgr.y (mgr.14838) 88 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:05.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:04 smithi153 bash[35780]: cluster 2024-02-10T09:39:03.466751+0000 mgr.y (mgr.14838) 88 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:07.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:06 smithi204 bash[35809]: cluster 2024-02-10T09:39:05.467350+0000 mgr.y (mgr.14838) 89 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-02-10T09:39:07.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:06 smithi153 bash[35780]: cluster 2024-02-10T09:39:05.467350+0000 mgr.y (mgr.14838) 89 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-02-10T09:39:07.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:06 smithi153 bash[39117]: cluster 2024-02-10T09:39:05.467350+0000 mgr.y (mgr.14838) 89 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-02-10T09:39:09.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:08 smithi204 bash[35809]: cluster 2024-02-10T09:39:07.467894+0000 mgr.y (mgr.14838) 90 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-02-10T09:39:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:08 smithi153 bash[35780]: cluster 2024-02-10T09:39:07.467894+0000 mgr.y (mgr.14838) 90 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-02-10T09:39:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:08 smithi153 bash[39117]: cluster 2024-02-10T09:39:07.467894+0000 mgr.y (mgr.14838) 90 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-02-10T09:39:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:10 smithi204 bash[35809]: cluster 2024-02-10T09:39:09.468410+0000 mgr.y (mgr.14838) 91 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-02-10T09:39:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:10 smithi153 bash[39117]: cluster 2024-02-10T09:39:09.468410+0000 mgr.y (mgr.14838) 91 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-02-10T09:39:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:10 smithi153 bash[35780]: cluster 2024-02-10T09:39:09.468410+0000 mgr.y (mgr.14838) 91 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-02-10T09:39:12.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:12.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:12.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:12.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 bash[52685]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:39:12.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:39:12.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:12.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:12 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:13.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:12 smithi204 bash[35809]: cluster 2024-02-10T09:39:11.469273+0000 mgr.y (mgr.14838) 92 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:13.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:12 smithi153 bash[39117]: cluster 2024-02-10T09:39:11.469273+0000 mgr.y (mgr.14838) 92 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:13.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:12 smithi153 bash[35780]: cluster 2024-02-10T09:39:11.469273+0000 mgr.y (mgr.14838) 92 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:39:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:39:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:14] "GET /metrics HTTP/1.1" 200 153229 "" "Prometheus/2.33.4" 2024-02-10T09:39:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:14 smithi204 bash[35809]: cluster 2024-02-10T09:39:13.469858+0000 mgr.y (mgr.14838) 93 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:15.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:14 smithi153 bash[35780]: cluster 2024-02-10T09:39:13.469858+0000 mgr.y (mgr.14838) 93 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:14 smithi153 bash[39117]: cluster 2024-02-10T09:39:13.469858+0000 mgr.y (mgr.14838) 93 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:17.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:16 smithi204 bash[35809]: cluster 2024-02-10T09:39:15.470568+0000 mgr.y (mgr.14838) 94 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:17.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:16 smithi153 bash[35780]: cluster 2024-02-10T09:39:15.470568+0000 mgr.y (mgr.14838) 94 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:17.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:16 smithi153 bash[39117]: cluster 2024-02-10T09:39:15.470568+0000 mgr.y (mgr.14838) 94 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:39:19.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:19 smithi204 bash[35809]: cluster 2024-02-10T09:39:17.471143+0000 mgr.y (mgr.14838) 95 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:19 smithi153 bash[35780]: cluster 2024-02-10T09:39:17.471143+0000 mgr.y (mgr.14838) 95 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:19 smithi153 bash[39117]: cluster 2024-02-10T09:39:17.471143+0000 mgr.y (mgr.14838) 95 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[39117]: cluster 2024-02-10T09:39:19.471564+0000 mgr.y (mgr.14838) 96 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:21.302 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[35780]: cluster 2024-02-10T09:39:19.471564+0000 mgr.y (mgr.14838) 96 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:21 smithi204 bash[35809]: cluster 2024-02-10T09:39:19.471564+0000 mgr.y (mgr.14838) 96 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:21.586 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[64224]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:39:21.586 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[64224]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:39:21.586 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[64224]: Error: 3 errors occurred while pulling: 2024-02-10T09:39:21.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[64224]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:39:21.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[64224]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:39:21.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 bash[64224]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:39:21.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Control process exited, code=exited status=125 2024-02-10T09:39:21.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:21.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:21 smithi153 systemd[1]: Failed to start Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:22.660 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:22.660 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:39:22.660 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:22.660 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:22.660 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:22.661 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[51909]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:39:22.662 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[51909]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:39:22.662 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[51909]: Error: 3 errors occurred while pulling: 2024-02-10T09:39:22.662 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[51909]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:39:22.662 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[51909]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:39:22.662 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[51909]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:39:23.012 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Control process exited, code=exited status=125 2024-02-10T09:39:23.012 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:39:23.012 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: Failed to start Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:23.012 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 bash[52703]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:39:23.012 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:39:23.012 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:23.012 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:22 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:23 smithi204 bash[35809]: cluster 2024-02-10T09:39:21.472483+0000 mgr.y (mgr.14838) 97 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:23.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:23 smithi153 bash[35780]: cluster 2024-02-10T09:39:21.472483+0000 mgr.y (mgr.14838) 97 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:23.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:23 smithi153 bash[39117]: cluster 2024-02-10T09:39:21.472483+0000 mgr.y (mgr.14838) 97 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T09:39:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:39:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:39:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:24] "GET /metrics HTTP/1.1" 200 153243 "" "Prometheus/2.33.4" 2024-02-10T09:39:25.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:25 smithi204 bash[35809]: cluster 2024-02-10T09:39:23.472995+0000 mgr.y (mgr.14838) 98 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:25 smithi153 bash[35780]: cluster 2024-02-10T09:39:23.472995+0000 mgr.y (mgr.14838) 98 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:25.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:25 smithi153 bash[39117]: cluster 2024-02-10T09:39:23.472995+0000 mgr.y (mgr.14838) 98 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:27.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:27 smithi204 bash[35809]: cluster 2024-02-10T09:39:25.473630+0000 mgr.y (mgr.14838) 99 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:27.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:27 smithi153 bash[35780]: cluster 2024-02-10T09:39:25.473630+0000 mgr.y (mgr.14838) 99 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:27.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:27 smithi153 bash[39117]: cluster 2024-02-10T09:39:25.473630+0000 mgr.y (mgr.14838) 99 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:29 smithi204 bash[35809]: cluster 2024-02-10T09:39:27.474063+0000 mgr.y (mgr.14838) 100 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:29.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:29 smithi153 bash[35780]: cluster 2024-02-10T09:39:27.474063+0000 mgr.y (mgr.14838) 100 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:29.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:29 smithi153 bash[39117]: cluster 2024-02-10T09:39:27.474063+0000 mgr.y (mgr.14838) 100 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:31 smithi204 bash[35809]: cluster 2024-02-10T09:39:29.474464+0000 mgr.y (mgr.14838) 101 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:31.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:31 smithi153 bash[35780]: cluster 2024-02-10T09:39:29.474464+0000 mgr.y (mgr.14838) 101 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:31 smithi153 bash[39117]: cluster 2024-02-10T09:39:29.474464+0000 mgr.y (mgr.14838) 101 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:32.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:31 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:32.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:31 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:39:32.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:31 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:32.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:31 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:32.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:31 smithi153 bash[64342]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:39:32.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:39:31 smithi153 bash[64342]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:39:33.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:33 smithi204 bash[35809]: cluster 2024-02-10T09:39:31.475259+0000 mgr.y (mgr.14838) 102 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:33.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:33.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:39:33.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:33.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:33.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:32 smithi204 bash[52730]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:39:33.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:39:32 smithi204 bash[52730]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:39:33.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:33.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:39:33.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 bash[52729]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:39:33.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:33.334 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:32 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:33.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:33 smithi153 bash[35780]: cluster 2024-02-10T09:39:31.475259+0000 mgr.y (mgr.14838) 102 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:33.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:33 smithi153 bash[39117]: cluster 2024-02-10T09:39:31.475259+0000 mgr.y (mgr.14838) 102 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:39:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:39:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:34] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:39:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:35 smithi204 bash[35809]: cluster 2024-02-10T09:39:33.475720+0000 mgr.y (mgr.14838) 103 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:35 smithi153 bash[35780]: cluster 2024-02-10T09:39:33.475720+0000 mgr.y (mgr.14838) 103 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:35.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:35 smithi153 bash[39117]: cluster 2024-02-10T09:39:33.475720+0000 mgr.y (mgr.14838) 103 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:37 smithi204 bash[35809]: cluster 2024-02-10T09:39:35.476398+0000 mgr.y (mgr.14838) 104 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:37.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:37 smithi153 bash[35780]: cluster 2024-02-10T09:39:35.476398+0000 mgr.y (mgr.14838) 104 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:37.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:37 smithi153 bash[39117]: cluster 2024-02-10T09:39:35.476398+0000 mgr.y (mgr.14838) 104 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:39.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:39 smithi204 bash[35809]: cluster 2024-02-10T09:39:37.476980+0000 mgr.y (mgr.14838) 105 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:39 smithi153 bash[35780]: cluster 2024-02-10T09:39:37.476980+0000 mgr.y (mgr.14838) 105 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:39.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:39 smithi153 bash[39117]: cluster 2024-02-10T09:39:37.476980+0000 mgr.y (mgr.14838) 105 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:41.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:41 smithi204 bash[35809]: cluster 2024-02-10T09:39:39.477352+0000 mgr.y (mgr.14838) 106 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:41.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:41 smithi153 bash[35780]: cluster 2024-02-10T09:39:39.477352+0000 mgr.y (mgr.14838) 106 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:41.414 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:41 smithi153 bash[39117]: cluster 2024-02-10T09:39:39.477352+0000 mgr.y (mgr.14838) 106 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:41.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:43 smithi153 bash[39117]: cluster 2024-02-10T09:39:41.478267+0000 mgr.y (mgr.14838) 107 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:43.301 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:43 smithi153 bash[35780]: cluster 2024-02-10T09:39:41.478267+0000 mgr.y (mgr.14838) 107 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:43 smithi204 bash[35809]: cluster 2024-02-10T09:39:41.478267+0000 mgr.y (mgr.14838) 107 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:39:43.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 bash[52759]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:43.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:43 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:43.710 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (7m) 2m ago 8m 25.9M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 error 2m ago 10m 27.0M - 6.6.2 87a51ecf0b1c 72f83803a230 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (12m) 2m ago 21m 382M - 16.2.14-521-g0e714d9a 1fbc92c859dc 46c6073a79be 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (3m) 2m ago 22m 415M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (22m) 2m ago 23m 126M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (21m) 2m ago 21m 136M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (21m) 2m ago 21m 135M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 error 2m ago 13m - - 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 error 2m ago 13m - - 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (20m) 2m ago 20m 42.3M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:39:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (20m) 2m ago 20m 42.9M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (20m) 2m ago 20m 41.5M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (19m) 2m ago 19m 45.0M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (19m) 2m ago 19m 45.7M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (19m) 2m ago 19m 47.1M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (18m) 2m ago 18m 45.0M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (18m) 2m ago 18m 45.3M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (7m) 2m ago 7m 57.4M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (10m) 2m ago 10m 33.8M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:39:43.712 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (10m) 2m ago 10m 35.1M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:39:44.060 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:39:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.701103+0000 mon.c (mon.1) 119 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.701471+0000 mon.c (mon.1) 120 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.701772+0000 mon.c (mon.1) 121 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.702048+0000 mon.c (mon.1) 122 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.702317+0000 mon.c (mon.1) 123 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.702568+0000 mon.c (mon.1) 124 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.324 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.704099+0000 mon.c (mon.1) 125 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.704378+0000 mon.c (mon.1) 126 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.704645+0000 mon.c (mon.1) 127 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.704908+0000 mon.c (mon.1) 128 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[35780]: audit 2024-02-10T09:39:43.705156+0000 mon.c (mon.1) 129 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.701103+0000 mon.c (mon.1) 119 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.701471+0000 mon.c (mon.1) 120 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.701772+0000 mon.c (mon.1) 121 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.702048+0000 mon.c (mon.1) 122 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.702317+0000 mon.c (mon.1) 123 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.702568+0000 mon.c (mon.1) 124 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.704099+0000 mon.c (mon.1) 125 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.325 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.704378+0000 mon.c (mon.1) 126 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.326 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.704645+0000 mon.c (mon.1) 127 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.326 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.704908+0000 mon.c (mon.1) 128 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.326 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:44 smithi153 bash[39117]: audit 2024-02-10T09:39:43.705156+0000 mon.c (mon.1) 129 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.701103+0000 mon.c (mon.1) 119 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.701471+0000 mon.c (mon.1) 120 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.701772+0000 mon.c (mon.1) 121 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.702048+0000 mon.c (mon.1) 122 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.702317+0000 mon.c (mon.1) 123 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.702568+0000 mon.c (mon.1) 124 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.704099+0000 mon.c (mon.1) 125 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.704378+0000 mon.c (mon.1) 126 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.704645+0000 mon.c (mon.1) 127 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.704908+0000 mon.c (mon.1) 128 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:44 smithi204 bash[35809]: audit 2024-02-10T09:39:43.705156+0000 mon.c (mon.1) 129 : audit [DBG] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:39:44.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:39:44.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:44] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:39:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: cluster 2024-02-10T09:39:43.478610+0000 mgr.y (mgr.14838) 108 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:43.699060+0000 mgr.y (mgr.14838) 109 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.954160+0000 mon.c (mon.1) 130 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.954365+0000 mon.c (mon.1) 131 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.954505+0000 mon.c (mon.1) 132 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.954525+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.954628+0000 mon.c (mon.1) 133 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.955042+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.955260+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:45 smithi204 bash[35809]: audit 2024-02-10T09:39:44.955481+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:39:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: cluster 2024-02-10T09:39:43.478610+0000 mgr.y (mgr.14838) 108 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:43.699060+0000 mgr.y (mgr.14838) 109 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:39:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.954160+0000 mon.c (mon.1) 130 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:39:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.954365+0000 mon.c (mon.1) 131 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.954505+0000 mon.c (mon.1) 132 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.954525+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.954628+0000 mon.c (mon.1) 133 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.955042+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.955260+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[35780]: audit 2024-02-10T09:39:44.955481+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: cluster 2024-02-10T09:39:43.478610+0000 mgr.y (mgr.14838) 108 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:43.699060+0000 mgr.y (mgr.14838) 109 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.954160+0000 mon.c (mon.1) 130 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.954365+0000 mon.c (mon.1) 131 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.954505+0000 mon.c (mon.1) 132 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.954525+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.954628+0000 mon.c (mon.1) 133 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:39:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.955042+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.955260+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:39:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:45 smithi153 bash[39117]: audit 2024-02-10T09:39:44.955481+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:39:46.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:39:46.315 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:39:46.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph -s' 2024-02-10T09:39:47.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:47 smithi153 bash[35780]: cluster 2024-02-10T09:39:45.479209+0000 mgr.y (mgr.14838) 110 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:47.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:47 smithi153 bash[35780]: audit 2024-02-10T09:39:46.313328+0000 mon.c (mon.1) 134 : audit [DBG] from='client.? 172.21.15.153:0/2283956130' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:39:47.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:47 smithi153 bash[39117]: cluster 2024-02-10T09:39:45.479209+0000 mgr.y (mgr.14838) 110 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:47.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:47 smithi153 bash[39117]: audit 2024-02-10T09:39:46.313328+0000 mon.c (mon.1) 134 : audit [DBG] from='client.? 172.21.15.153:0/2283956130' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:39:47.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:47 smithi204 bash[35809]: cluster 2024-02-10T09:39:45.479209+0000 mgr.y (mgr.14838) 110 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:47.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:47 smithi204 bash[35809]: audit 2024-02-10T09:39:46.313328+0000 mon.c (mon.1) 134 : audit [DBG] from='client.? 172.21.15.153:0/2283956130' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:39:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:48 smithi153 bash[35780]: cluster 2024-02-10T09:39:47.479551+0000 mgr.y (mgr.14838) 111 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:48 smithi153 bash[39117]: cluster 2024-02-10T09:39:47.479551+0000 mgr.y (mgr.14838) 111 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:48.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:48 smithi204 bash[35809]: cluster 2024-02-10T09:39:47.479551+0000 mgr.y (mgr.14838) 111 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:48.975 INFO:teuthology.orchestra.run.smithi153.stdout: cluster: 2024-02-10T09:39:48.975 INFO:teuthology.orchestra.run.smithi153.stdout: id: 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:39:48.975 INFO:teuthology.orchestra.run.smithi153.stdout: health: HEALTH_WARN 2024-02-10T09:39:48.975 INFO:teuthology.orchestra.run.smithi153.stdout: 3 failed cephadm daemon(s) 2024-02-10T09:39:48.975 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: services: 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: mgr: y(active, since 3m), standbys: x 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: rgw: 2 daemons active (r.z.smithi153.zyveqt, r.z.smithi204.kmhqqg) 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: task status: 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: data: 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: pools: 5 pools, 129 pgs 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: objects: 202 objects, 7.0 KiB 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: usage: 8.0 GiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:48.976 INFO:teuthology.orchestra.run.smithi153.stdout: pgs: 129 active+clean 2024-02-10T09:39:48.977 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-02-10T09:39:49.269 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:49 smithi153 bash[35780]: audit 2024-02-10T09:39:48.974877+0000 mon.c (mon.1) 135 : audit [DBG] from='client.? 172.21.15.153:0/3116811815' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:39:49.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:49 smithi153 bash[39117]: audit 2024-02-10T09:39:48.974877+0000 mon.c (mon.1) 135 : audit [DBG] from='client.? 172.21.15.153:0/3116811815' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:39:49.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:49 smithi204 bash[35809]: audit 2024-02-10T09:39:48.974877+0000 mon.c (mon.1) 135 : audit [DBG] from='client.? 172.21.15.153:0/3116811815' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:39:49.586 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-10T09:39:50.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:50 smithi153 bash[35780]: cluster 2024-02-10T09:39:49.480090+0000 mgr.y (mgr.14838) 112 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:50.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:50 smithi153 bash[39117]: cluster 2024-02-10T09:39:49.480090+0000 mgr.y (mgr.14838) 112 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:50.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:50 smithi204 bash[35809]: cluster 2024-02-10T09:39:49.480090+0000 mgr.y (mgr.14838) 112 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:51.485 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:39:51.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:51 smithi204 bash[35809]: audit 2024-02-10T09:39:51.474225+0000 mon.c (mon.1) 136 : audit [DBG] from='client.? 172.21.15.153:0/2469853632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:39:51.850 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:51 smithi153 bash[35780]: audit 2024-02-10T09:39:51.474225+0000 mon.c (mon.1) 136 : audit [DBG] from='client.? 172.21.15.153:0/2469853632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:39:51.850 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:51 smithi153 bash[39117]: audit 2024-02-10T09:39:51.474225+0000 mon.c (mon.1) 136 : audit [DBG] from='client.? 172.21.15.153:0/2469853632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:39:52.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph mgr fail' 2024-02-10T09:39:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:52 smithi153 bash[39117]: cluster 2024-02-10T09:39:51.480670+0000 mgr.y (mgr.14838) 113 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:53.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:52 smithi153 bash[35780]: cluster 2024-02-10T09:39:51.480670+0000 mgr.y (mgr.14838) 113 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:52 smithi204 bash[35809]: cluster 2024-02-10T09:39:51.480670+0000 mgr.y (mgr.14838) 113 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:53.608 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:39:53.608 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:39:53.608 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:53 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:53.608 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-02-10T09:39:53.608 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:53 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:39:53.609 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:39:53 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:39:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:39:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:39:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:39:54] "GET /metrics HTTP/1.1" 200 153243 "" "Prometheus/2.33.4" 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:54.860+0000 7f113dcef700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ignoring --setuser ceph since I am not root 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ignoring --setgroup ceph since I am not root 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:54.907+0000 7f132b175700 1 -- 172.21.15.153:0/1016787879 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592361024e0 con 0x5592360d9400 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:55.038+0000 7f133b353680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:55.120+0000 7f133b353680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: cluster 2024-02-10T09:39:53.481078+0000 mgr.y (mgr.14838) 114 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:55.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: audit 2024-02-10T09:39:53.958917+0000 mon.c (mon.1) 137 : audit [INF] from='client.? 172.21.15.153:0/1005203634' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: audit 2024-02-10T09:39:53.959210+0000 mon.a (mon.0) 804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: cluster 2024-02-10T09:39:53.963660+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: cluster 2024-02-10T09:39:53.974437+0000 mon.a (mon.0) 806 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: audit 2024-02-10T09:39:53.974479+0000 mon.c (mon.1) 138 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: audit 2024-02-10T09:39:53.974774+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: audit 2024-02-10T09:39:53.975345+0000 mon.c (mon.1) 139 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[39117]: audit 2024-02-10T09:39:53.975554+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: cluster 2024-02-10T09:39:53.481078+0000 mgr.y (mgr.14838) 114 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: audit 2024-02-10T09:39:53.958917+0000 mon.c (mon.1) 137 : audit [INF] from='client.? 172.21.15.153:0/1005203634' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: audit 2024-02-10T09:39:53.959210+0000 mon.a (mon.0) 804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:39:55.123 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: cluster 2024-02-10T09:39:53.963660+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-10T09:39:55.124 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: cluster 2024-02-10T09:39:53.974437+0000 mon.a (mon.0) 806 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:39:55.124 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: audit 2024-02-10T09:39:53.974479+0000 mon.c (mon.1) 138 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:39:55.124 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: audit 2024-02-10T09:39:53.974774+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:39:55.124 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: audit 2024-02-10T09:39:53.975345+0000 mon.c (mon.1) 139 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:39:55.124 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:54 smithi153 bash[35780]: audit 2024-02-10T09:39:53.975554+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:39:55.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: cluster 2024-02-10T09:39:53.481078+0000 mgr.y (mgr.14838) 114 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 5.9 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: audit 2024-02-10T09:39:53.958917+0000 mon.c (mon.1) 137 : audit [INF] from='client.? 172.21.15.153:0/1005203634' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: audit 2024-02-10T09:39:53.959210+0000 mon.a (mon.0) 804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: cluster 2024-02-10T09:39:53.963660+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: cluster 2024-02-10T09:39:53.974437+0000 mon.a (mon.0) 806 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: audit 2024-02-10T09:39:53.974479+0000 mon.c (mon.1) 138 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: audit 2024-02-10T09:39:53.974774+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: audit 2024-02-10T09:39:53.975345+0000 mon.c (mon.1) 139 : audit [INF] from='mgr.14838 172.21.15.153:0/2812068192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:39:55.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:54 smithi204 bash[35809]: audit 2024-02-10T09:39:53.975554+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:39:55.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 180' 2024-02-10T09:39:55.843 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:55.490+0000 7f133b353680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:39:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[39117]: audit 2024-02-10T09:39:54.860410+0000 mon.a (mon.0) 809 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:39:56.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[39117]: cluster 2024-02-10T09:39:54.860494+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e31: x(active, starting, since 0.900743s), standbys: y 2024-02-10T09:39:56.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[39117]: audit 2024-02-10T09:39:54.966513+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:39:56.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[39117]: cluster 2024-02-10T09:39:54.966576+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-10T09:39:56.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[35780]: audit 2024-02-10T09:39:54.860410+0000 mon.a (mon.0) 809 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:39:56.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[35780]: cluster 2024-02-10T09:39:54.860494+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e31: x(active, starting, since 0.900743s), standbys: y 2024-02-10T09:39:56.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[35780]: audit 2024-02-10T09:39:54.966513+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:39:56.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:39:55 smithi153 bash[35780]: cluster 2024-02-10T09:39:54.966576+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-10T09:39:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:55 smithi204 bash[35809]: audit 2024-02-10T09:39:54.860410+0000 mon.a (mon.0) 809 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T09:39:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:55 smithi204 bash[35809]: cluster 2024-02-10T09:39:54.860494+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e31: x(active, starting, since 0.900743s), standbys: y 2024-02-10T09:39:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:55 smithi204 bash[35809]: audit 2024-02-10T09:39:54.966513+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14838 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:39:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:39:55 smithi204 bash[35809]: cluster 2024-02-10T09:39:54.966576+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-10T09:39:56.477 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:56.278+0000 7f133b353680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:39:56.477 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:56.365+0000 7f133b353680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:39:56.791 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:56.582+0000 7f133b353680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:39:57.847 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:57.548+0000 7f133b353680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:39:57.847 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:57.676+0000 7f133b353680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:39:57.847 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:57.755+0000 7f133b353680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:39:57.847 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:57.846+0000 7f133b353680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:39:58.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:58.044+0000 7f133b353680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:39:58.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:58.325+0000 7f133b353680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:39:58.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:58.424+0000 7f133b353680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:39:59.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:59.152+0000 7f133b353680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:39:59.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:59.235+0000 7f133b353680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:39:59.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:59.328+0000 7f133b353680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:39:59.768 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:59.505+0000 7f133b353680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:40:00.035 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:59.767+0000 7f133b353680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:40:00.036 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:39:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:39:59.900+0000 7f133b353680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:40:00.036 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:40:00.034+0000 7f133b353680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:40:00.294 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:40:00.207+0000 7f133b353680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:40:00.293+0000 7f133b353680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:40:00] ENGINE Bus STARTING 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: CherryPy Checker: 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: The Application mounted at '' has an empty config. 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:40:00] ENGINE Serving on http://:::9283 2024-02-10T09:40:00.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:40:00] ENGINE Bus STARTED 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[35780]: cluster 2024-02-10T09:40:00.307743+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[35780]: cluster 2024-02-10T09:40:00.307838+0000 mon.a (mon.0) 814 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[35780]: audit 2024-02-10T09:40:00.308424+0000 mon.b (mon.2) 262 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[35780]: audit 2024-02-10T09:40:00.309046+0000 mon.b (mon.2) 263 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[35780]: audit 2024-02-10T09:40:00.311234+0000 mon.b (mon.2) 264 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[35780]: audit 2024-02-10T09:40:00.311994+0000 mon.b (mon.2) 265 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[39117]: cluster 2024-02-10T09:40:00.307743+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:40:00.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[39117]: cluster 2024-02-10T09:40:00.307838+0000 mon.a (mon.0) 814 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:40:00.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[39117]: audit 2024-02-10T09:40:00.308424+0000 mon.b (mon.2) 262 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T09:40:00.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[39117]: audit 2024-02-10T09:40:00.309046+0000 mon.b (mon.2) 263 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:40:00.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[39117]: audit 2024-02-10T09:40:00.311234+0000 mon.b (mon.2) 264 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T09:40:00.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:00 smithi153 bash[39117]: audit 2024-02-10T09:40:00.311994+0000 mon.b (mon.2) 265 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:40:00.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[35809]: cluster 2024-02-10T09:40:00.307743+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon y restarted 2024-02-10T09:40:00.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[35809]: cluster 2024-02-10T09:40:00.307838+0000 mon.a (mon.0) 814 : cluster [DBG] Standby manager daemon y started 2024-02-10T09:40:00.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[35809]: audit 2024-02-10T09:40:00.308424+0000 mon.b (mon.2) 262 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T09:40:00.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[35809]: audit 2024-02-10T09:40:00.309046+0000 mon.b (mon.2) 263 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:40:00.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[35809]: audit 2024-02-10T09:40:00.311234+0000 mon.b (mon.2) 264 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T09:40:00.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[35809]: audit 2024-02-10T09:40:00.311994+0000 mon.b (mon.2) 265 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:40:01.134 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[44646]: [10/Feb/2024:09:40:00] ENGINE Bus STOPPING 2024-02-10T09:40:01.135 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[44646]: [10/Feb/2024:09:40:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:40:01.135 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:00 smithi204 bash[44646]: [10/Feb/2024:09:40:00] ENGINE Bus STOPPED 2024-02-10T09:40:01.135 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: [10/Feb/2024:09:40:01] ENGINE Bus STARTING 2024-02-10T09:40:01.135 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: Warning: Permanently added '172.21.15.153' (ECDSA) to the list of known hosts. 2024-02-10T09:40:01.135 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: Warning: Permanently added '172.21.15.204' (ECDSA) to the list of known hosts. 2024-02-10T09:40:01.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: CherryPy Checker: 2024-02-10T09:40:01.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: The Application mounted at '' has an empty config. 2024-02-10T09:40:01.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: [10/Feb/2024:09:40:01] ENGINE Serving on http://:::9283 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[44646]: [10/Feb/2024:09:40:01] ENGINE Bus STARTED 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: cluster 2024-02-10T09:40:00.363549+0000 mon.a (mon.0) 815 : cluster [DBG] mgrmap e32: x(active, starting, since 6s), standbys: y 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.367584+0000 mon.b (mon.2) 266 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.367683+0000 mon.b (mon.2) 267 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.862787+0000 mon.b (mon.2) 268 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.863050+0000 mon.b (mon.2) 269 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:40:01.582 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.863287+0000 mon.b (mon.2) 270 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.864926+0000 mon.b (mon.2) 271 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.865228+0000 mon.b (mon.2) 272 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.865476+0000 mon.b (mon.2) 273 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.865725+0000 mon.b (mon.2) 274 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.865947+0000 mon.b (mon.2) 275 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.866188+0000 mon.b (mon.2) 276 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.866412+0000 mon.b (mon.2) 277 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.866635+0000 mon.b (mon.2) 278 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.868019+0000 mon.b (mon.2) 279 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.868265+0000 mon.b (mon.2) 280 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.868813+0000 mon.b (mon.2) 281 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.869666+0000 mon.b (mon.2) 282 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.870480+0000 mon.b (mon.2) 283 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.870790+0000 mon.b (mon.2) 284 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:40:01.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.871096+0000 mon.b (mon.2) 285 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.871418+0000 mon.b (mon.2) 286 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.871826+0000 mon.b (mon.2) 287 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.872154+0000 mon.b (mon.2) 288 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.872461+0000 mon.b (mon.2) 289 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.872772+0000 mon.b (mon.2) 290 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.873068+0000 mon.b (mon.2) 291 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.873372+0000 mon.b (mon.2) 292 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.873675+0000 mon.b (mon.2) 293 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.873963+0000 mon.b (mon.2) 294 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.874281+0000 mon.b (mon.2) 295 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.874563+0000 mon.b (mon.2) 296 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.874842+0000 mon.b (mon.2) 297 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.875119+0000 mon.b (mon.2) 298 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:40:01.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.875418+0000 mon.b (mon.2) 299 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.875695+0000 mon.b (mon.2) 300 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.875973+0000 mon.b (mon.2) 301 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.876259+0000 mon.b (mon.2) 302 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.876557+0000 mon.b (mon.2) 303 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.876830+0000 mon.b (mon.2) 304 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.877108+0000 mon.b (mon.2) 305 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.877423+0000 mon.b (mon.2) 306 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:00.877714+0000 mon.b (mon.2) 307 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: cluster 2024-02-10T09:40:00.906571+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon x is now available 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:01.009832+0000 mon.b (mon.2) 308 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:01.011205+0000 mon.b (mon.2) 309 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:01.056326+0000 mon.b (mon.2) 310 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:01.056879+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:01.066770+0000 mon.b (mon.2) 311 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:40:01.585 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:01 smithi204 bash[35809]: audit 2024-02-10T09:40:01.067200+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:40:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: cluster 2024-02-10T09:40:00.363549+0000 mon.a (mon.0) 815 : cluster [DBG] mgrmap e32: x(active, starting, since 6s), standbys: y 2024-02-10T09:40:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.367584+0000 mon.b (mon.2) 266 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.367683+0000 mon.b (mon.2) 267 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.862787+0000 mon.b (mon.2) 268 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.863050+0000 mon.b (mon.2) 269 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.863287+0000 mon.b (mon.2) 270 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.864926+0000 mon.b (mon.2) 271 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.865228+0000 mon.b (mon.2) 272 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.865476+0000 mon.b (mon.2) 273 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.865725+0000 mon.b (mon.2) 274 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.865947+0000 mon.b (mon.2) 275 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.866188+0000 mon.b (mon.2) 276 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.866412+0000 mon.b (mon.2) 277 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.866635+0000 mon.b (mon.2) 278 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.868019+0000 mon.b (mon.2) 279 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:40:01.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.868265+0000 mon.b (mon.2) 280 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.868813+0000 mon.b (mon.2) 281 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.869666+0000 mon.b (mon.2) 282 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.870480+0000 mon.b (mon.2) 283 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.870790+0000 mon.b (mon.2) 284 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.871096+0000 mon.b (mon.2) 285 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.871418+0000 mon.b (mon.2) 286 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.871826+0000 mon.b (mon.2) 287 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.872154+0000 mon.b (mon.2) 288 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.872461+0000 mon.b (mon.2) 289 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.872772+0000 mon.b (mon.2) 290 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.873068+0000 mon.b (mon.2) 291 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.873372+0000 mon.b (mon.2) 292 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.873675+0000 mon.b (mon.2) 293 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.873963+0000 mon.b (mon.2) 294 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:40:01.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.874281+0000 mon.b (mon.2) 295 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.874563+0000 mon.b (mon.2) 296 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.874842+0000 mon.b (mon.2) 297 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.875119+0000 mon.b (mon.2) 298 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.875418+0000 mon.b (mon.2) 299 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: cluster 2024-02-10T09:40:00.363549+0000 mon.a (mon.0) 815 : cluster [DBG] mgrmap e32: x(active, starting, since 6s), standbys: y 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.367584+0000 mon.b (mon.2) 266 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.367683+0000 mon.b (mon.2) 267 : audit [DBG] from='mgr.? 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.862787+0000 mon.b (mon.2) 268 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.863050+0000 mon.b (mon.2) 269 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:40:01.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.863287+0000 mon.b (mon.2) 270 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.864926+0000 mon.b (mon.2) 271 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.865228+0000 mon.b (mon.2) 272 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.865476+0000 mon.b (mon.2) 273 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.865725+0000 mon.b (mon.2) 274 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.865947+0000 mon.b (mon.2) 275 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.866188+0000 mon.b (mon.2) 276 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.866412+0000 mon.b (mon.2) 277 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.866635+0000 mon.b (mon.2) 278 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.868019+0000 mon.b (mon.2) 279 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.868265+0000 mon.b (mon.2) 280 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.868813+0000 mon.b (mon.2) 281 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.869666+0000 mon.b (mon.2) 282 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.870480+0000 mon.b (mon.2) 283 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.870790+0000 mon.b (mon.2) 284 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:40:01.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.871096+0000 mon.b (mon.2) 285 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.871418+0000 mon.b (mon.2) 286 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.871826+0000 mon.b (mon.2) 287 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.872154+0000 mon.b (mon.2) 288 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.872461+0000 mon.b (mon.2) 289 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.872772+0000 mon.b (mon.2) 290 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.873068+0000 mon.b (mon.2) 291 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.873372+0000 mon.b (mon.2) 292 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.873675+0000 mon.b (mon.2) 293 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.873963+0000 mon.b (mon.2) 294 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.874281+0000 mon.b (mon.2) 295 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.874563+0000 mon.b (mon.2) 296 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.874842+0000 mon.b (mon.2) 297 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.875119+0000 mon.b (mon.2) 298 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.875418+0000 mon.b (mon.2) 299 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:40:01.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.875695+0000 mon.b (mon.2) 300 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.875973+0000 mon.b (mon.2) 301 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.876259+0000 mon.b (mon.2) 302 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.876557+0000 mon.b (mon.2) 303 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.876830+0000 mon.b (mon.2) 304 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.877108+0000 mon.b (mon.2) 305 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.877423+0000 mon.b (mon.2) 306 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:00.877714+0000 mon.b (mon.2) 307 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: cluster 2024-02-10T09:40:00.906571+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon x is now available 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:01.009832+0000 mon.b (mon.2) 308 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:01.011205+0000 mon.b (mon.2) 309 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:01.056326+0000 mon.b (mon.2) 310 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:01.056879+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:40:01.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:01.066770+0000 mon.b (mon.2) 311 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[39117]: audit 2024-02-10T09:40:01.067200+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.875695+0000 mon.b (mon.2) 300 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.875973+0000 mon.b (mon.2) 301 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.876259+0000 mon.b (mon.2) 302 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.876557+0000 mon.b (mon.2) 303 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.876830+0000 mon.b (mon.2) 304 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.877108+0000 mon.b (mon.2) 305 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.877423+0000 mon.b (mon.2) 306 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:00.877714+0000 mon.b (mon.2) 307 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: cluster 2024-02-10T09:40:00.906571+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon x is now available 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:01.009832+0000 mon.b (mon.2) 308 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:01.011205+0000 mon.b (mon.2) 309 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:01.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:01.056326+0000 mon.b (mon.2) 310 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:40:01.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:01.056879+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:40:01.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:01.066770+0000 mon.b (mon.2) 311 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:40:01.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:01 smithi153 bash[35780]: audit 2024-02-10T09:40:01.067200+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T09:40:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: cluster 2024-02-10T09:40:01.387973+0000 mon.a (mon.0) 819 : cluster [DBG] mgrmap e33: x(active, since 7s), standbys: y 2024-02-10T09:40:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: cluster 2024-02-10T09:40:01.393915+0000 mgr.x (mgr.34700) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: cluster 2024-02-10T09:40:01.387973+0000 mon.a (mon.0) 819 : cluster [DBG] mgrmap e33: x(active, since 7s), standbys: y 2024-02-10T09:40:02.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: cluster 2024-02-10T09:40:01.393915+0000 mgr.x (mgr.34700) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:02.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.483657+0000 mon.b (mon.2) 312 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.483657+0000 mon.b (mon.2) 312 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.484285+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.484664+0000 mon.b (mon.2) 313 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.484285+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.485129+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.489857+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:02.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.484664+0000 mon.b (mon.2) 313 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.490848+0000 mon.b (mon.2) 314 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:40:02.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.494910+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:02.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[39117]: audit 2024-02-10T09:40:01.495767+0000 mon.b (mon.2) 315 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:40:02.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.485129+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.489857+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:02.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.490848+0000 mon.b (mon.2) 314 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:40:02.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.494910+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:02.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:02 smithi153 bash[35780]: audit 2024-02-10T09:40:01.495767+0000 mon.b (mon.2) 315 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:40:02.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: cluster 2024-02-10T09:40:01.387973+0000 mon.a (mon.0) 819 : cluster [DBG] mgrmap e33: x(active, since 7s), standbys: y 2024-02-10T09:40:02.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: cluster 2024-02-10T09:40:01.393915+0000 mgr.x (mgr.34700) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:02.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.483657+0000 mon.b (mon.2) 312 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.484285+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.484664+0000 mon.b (mon.2) 313 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.485129+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:02.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.489857+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 132959436, \"cpu_percentage\": \"1.28%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049578Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 435159040, \"cpu_percentage\": \"13.96%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:36:49.049758Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 141557760, \"cpu_percentage\": \"0.71%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049839Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 44396707, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049906Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45078282, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.049971Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 43536875, \"cpu_percentage\": \"0.39%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050034Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47269806, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050097Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050159Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 35494297, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.050199Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27242004, \"cpu_percentage\": \"0.13%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:36:49.050278Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:02.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.490848+0000 mon.b (mon.2) 314 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:40:02.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.494910+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 142920908, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840231Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.86%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840379Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48003809, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840451Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 49482301, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840516Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47227863, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840580Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47573893, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840642Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840720Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 28395438, \"cpu_percentage\": \"0.44%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840776Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 36846960, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:36:49.840840Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 60209233, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:36:49.840903Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:02.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:02 smithi204 bash[35809]: audit 2024-02-10T09:40:01.495767+0000 mon.b (mon.2) 315 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:40:03.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:03 smithi204 bash[35809]: cluster 2024-02-10T09:40:02.384019+0000 mon.a (mon.0) 824 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:40:03.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:03 smithi204 bash[35809]: cluster 2024-02-10T09:40:02.384072+0000 mon.a (mon.0) 825 : cluster [INF] Cluster is now healthy 2024-02-10T09:40:03.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:03 smithi204 bash[35809]: cluster 2024-02-10T09:40:02.394743+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e34: x(active, since 8s), standbys: y 2024-02-10T09:40:03.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:03 smithi204 bash[35809]: cluster 2024-02-10T09:40:02.868277+0000 mgr.x (mgr.34700) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[39117]: cluster 2024-02-10T09:40:02.384019+0000 mon.a (mon.0) 824 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[39117]: cluster 2024-02-10T09:40:02.384072+0000 mon.a (mon.0) 825 : cluster [INF] Cluster is now healthy 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[39117]: cluster 2024-02-10T09:40:02.394743+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e34: x(active, since 8s), standbys: y 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[39117]: cluster 2024-02-10T09:40:02.868277+0000 mgr.x (mgr.34700) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[35780]: cluster 2024-02-10T09:40:02.384019+0000 mon.a (mon.0) 824 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[35780]: cluster 2024-02-10T09:40:02.384072+0000 mon.a (mon.0) 825 : cluster [INF] Cluster is now healthy 2024-02-10T09:40:03.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[35780]: cluster 2024-02-10T09:40:02.394743+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e34: x(active, since 8s), standbys: y 2024-02-10T09:40:03.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:03 smithi153 bash[35780]: cluster 2024-02-10T09:40:02.868277+0000 mgr.x (mgr.34700) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:03] "GET /metrics HTTP/1.1" 200 32892 "" "Prometheus/2.33.4" 2024-02-10T09:40:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:40:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: cluster 2024-02-10T09:40:04.868590+0000 mgr.x (mgr.34700) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: cluster 2024-02-10T09:40:04.868590+0000 mgr.x (mgr.34700) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.107080+0000 mon.b (mon.2) 316 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.107080+0000 mon.b (mon.2) 316 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:06.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.108209+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:06.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.111820+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:06.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.380977+0000 mon.b (mon.2) 317 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.381355+0000 mon.b (mon.2) 318 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.381676+0000 mon.b (mon.2) 319 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.381926+0000 mon.b (mon.2) 320 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.382176+0000 mon.b (mon.2) 321 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.382427+0000 mon.b (mon.2) 322 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.382670+0000 mon.b (mon.2) 323 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.382923+0000 mon.b (mon.2) 324 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[35780]: audit 2024-02-10T09:40:05.383184+0000 mon.b (mon.2) 325 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.108209+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.111820+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.380977+0000 mon.b (mon.2) 317 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.381355+0000 mon.b (mon.2) 318 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.381676+0000 mon.b (mon.2) 319 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.381926+0000 mon.b (mon.2) 320 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.382176+0000 mon.b (mon.2) 321 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.382427+0000 mon.b (mon.2) 322 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.382670+0000 mon.b (mon.2) 323 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.382923+0000 mon.b (mon.2) 324 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:05 smithi153 bash[39117]: audit 2024-02-10T09:40:05.383184+0000 mon.b (mon.2) 325 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.233 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: cluster 2024-02-10T09:40:04.868590+0000 mgr.x (mgr.34700) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.107080+0000 mon.b (mon.2) 316 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.108209+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.111820+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 181718220, \"cpu_percentage\": \"1.26%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105144Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 400975462, \"cpu_percentage\": \"7.27%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:40:05.105307Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 153616384, \"cpu_percentage\": \"0.73%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105381Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45602570, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105461Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46378516, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105523Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 44774195, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105583Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47961866, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105645Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105705Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 36511416, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.105742Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27252490, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:40:05.105803Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.105927Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.380977+0000 mon.b (mon.2) 317 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.381355+0000 mon.b (mon.2) 318 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.381676+0000 mon.b (mon.2) 319 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.381926+0000 mon.b (mon.2) 320 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.382176+0000 mon.b (mon.2) 321 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.382427+0000 mon.b (mon.2) 322 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.382670+0000 mon.b (mon.2) 323 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.382923+0000 mon.b (mon.2) 324 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:06.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:05 smithi204 bash[35809]: audit 2024-02-10T09:40:05.383184+0000 mon.b (mon.2) 325 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:05.951368+0000 mon.b (mon.2) 326 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:05.952052+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:05.955203+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.224234+0000 mon.b (mon.2) 327 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.224576+0000 mon.b (mon.2) 328 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.224831+0000 mon.b (mon.2) 329 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.225095+0000 mon.b (mon.2) 330 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.225372+0000 mon.b (mon.2) 331 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.225620+0000 mon.b (mon.2) 332 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.225872+0000 mon.b (mon.2) 333 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.226115+0000 mon.b (mon.2) 334 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.226389+0000 mon.b (mon.2) 335 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.227640+0000 mon.b (mon.2) 336 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.228243+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:07.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:06 smithi204 bash[35809]: audit 2024-02-10T09:40:06.231480+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:40:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:05.951368+0000 mon.b (mon.2) 326 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:05.951368+0000 mon.b (mon.2) 326 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:05.952052+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:05.955203+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.224234+0000 mon.b (mon.2) 327 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.224576+0000 mon.b (mon.2) 328 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.224831+0000 mon.b (mon.2) 329 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.225095+0000 mon.b (mon.2) 330 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.225372+0000 mon.b (mon.2) 331 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.225620+0000 mon.b (mon.2) 332 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.225872+0000 mon.b (mon.2) 333 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.226115+0000 mon.b (mon.2) 334 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.226389+0000 mon.b (mon.2) 335 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.227640+0000 mon.b (mon.2) 336 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.228243+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:07.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[39117]: audit 2024-02-10T09:40:06.231480+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:05.952052+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:05.955203+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 155608678, \"cpu_percentage\": \"0.86%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949439Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432852172, \"cpu_percentage\": \"3.93%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949596Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49912217, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949668Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50740592, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949733Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48496640, \"cpu_percentage\": \"0.48%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949796Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47689236, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949857Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949921Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37014732, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.949960Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37769707, \"cpu_percentage\": \"0.15%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:40:05.950021Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 66972549, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:40:05.950085Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:40:05.950226Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.224234+0000 mon.b (mon.2) 327 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.224576+0000 mon.b (mon.2) 328 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.224831+0000 mon.b (mon.2) 329 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.225095+0000 mon.b (mon.2) 330 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.225372+0000 mon.b (mon.2) 331 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.225620+0000 mon.b (mon.2) 332 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.225872+0000 mon.b (mon.2) 333 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.226115+0000 mon.b (mon.2) 334 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:40:07.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.226389+0000 mon.b (mon.2) 335 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:40:07.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.227640+0000 mon.b (mon.2) 336 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:07.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.228243+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:07.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:06 smithi153 bash[35780]: audit 2024-02-10T09:40:06.231480+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:40:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:07 smithi153 bash[35780]: cephadm 2024-02-10T09:40:06.235494+0000 mgr.x (mgr.34700) 5 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:40:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:07 smithi153 bash[35780]: cephadm 2024-02-10T09:40:06.268781+0000 mgr.x (mgr.34700) 6 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:40:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:07 smithi153 bash[35780]: cluster 2024-02-10T09:40:06.953478+0000 mon.a (mon.0) 833 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:40:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:07 smithi153 bash[39117]: cephadm 2024-02-10T09:40:06.235494+0000 mgr.x (mgr.34700) 5 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:40:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:07 smithi153 bash[39117]: cephadm 2024-02-10T09:40:06.268781+0000 mgr.x (mgr.34700) 6 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:40:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:07 smithi153 bash[39117]: cluster 2024-02-10T09:40:06.953478+0000 mon.a (mon.0) 833 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:40:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:07 smithi204 bash[35809]: cephadm 2024-02-10T09:40:06.235494+0000 mgr.x (mgr.34700) 5 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:40:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:07 smithi204 bash[35809]: cephadm 2024-02-10T09:40:06.268781+0000 mgr.x (mgr.34700) 6 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:40:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:07 smithi204 bash[35809]: cluster 2024-02-10T09:40:06.953478+0000 mon.a (mon.0) 833 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:40:08.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:40:08 smithi153 bash[64342]: time="2024-02-10T09:40:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:40:09.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[35809]: audit 2024-02-10T09:40:08.930542+0000 mon.b (mon.2) 337 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: debug 2024-02-10T09:40:08.927+0000 7fc22683d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: main() 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:40:09.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:09.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:09.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:09.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[53807]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:08 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:09.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:09.085 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: main() 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:09.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:09.087 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:08 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:08 smithi153 bash[35780]: audit 2024-02-10T09:40:08.930542+0000 mon.b (mon.2) 337 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:08 smithi153 bash[39117]: audit 2024-02-10T09:40:08.930542+0000 mon.b (mon.2) 337 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:40:09.686 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[52730]: time="2024-02-10T09:40:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:40:10.265 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cluster 2024-02-10T09:40:08.227934+0000 mgr.x (mgr.34700) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.265 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cephadm 2024-02-10T09:40:08.928505+0000 mgr.x (mgr.34700) 8 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.265 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.265 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:40:10.265 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: main() 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:10.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:40:10.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: main() 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cluster 2024-02-10T09:40:08.932397+0000 mgr.x (mgr.34700) 9 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cluster 2024-02-10T09:40:08.932591+0000 mgr.x (mgr.34700) 10 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: audit 2024-02-10T09:40:08.932982+0000 mon.b (mon.2) 338 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:10.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cluster 2024-02-10T09:40:08.933616+0000 mon.a (mon.0) 834 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:40:10.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cluster 2024-02-10T09:40:08.933666+0000 mon.a (mon.0) 835 : cluster [INF] Cluster is now healthy 2024-02-10T09:40:10.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: audit 2024-02-10T09:40:08.934652+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:10.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: audit 2024-02-10T09:40:08.942880+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:40:10.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cephadm 2024-02-10T09:40:08.947683+0000 mgr.x (mgr.34700) 11 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:40:10.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cephadm 2024-02-10T09:40:08.954033+0000 mgr.x (mgr.34700) 12 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:40:10.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:09 smithi204 bash[35809]: cluster 2024-02-10T09:40:09.934916+0000 mon.a (mon.0) 838 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:40:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cluster 2024-02-10T09:40:08.227934+0000 mgr.x (mgr.34700) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cephadm 2024-02-10T09:40:08.928505+0000 mgr.x (mgr.34700) 8 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: main() 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:10.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:40:10.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cluster 2024-02-10T09:40:08.227934+0000 mgr.x (mgr.34700) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cephadm 2024-02-10T09:40:08.928505+0000 mgr.x (mgr.34700) 8 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:10.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: main() 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:10.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:40:10.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: main() 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:10.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cluster 2024-02-10T09:40:08.932397+0000 mgr.x (mgr.34700) 9 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cluster 2024-02-10T09:40:08.932591+0000 mgr.x (mgr.34700) 10 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: audit 2024-02-10T09:40:08.932982+0000 mon.b (mon.2) 338 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cluster 2024-02-10T09:40:08.933616+0000 mon.a (mon.0) 834 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cluster 2024-02-10T09:40:08.933666+0000 mon.a (mon.0) 835 : cluster [INF] Cluster is now healthy 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: audit 2024-02-10T09:40:08.934652+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:40:10.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: main() 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:10.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cluster 2024-02-10T09:40:08.932397+0000 mgr.x (mgr.34700) 9 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cluster 2024-02-10T09:40:08.932591+0000 mgr.x (mgr.34700) 10 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: audit 2024-02-10T09:40:08.932982+0000 mon.b (mon.2) 338 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cluster 2024-02-10T09:40:08.933616+0000 mon.a (mon.0) 834 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cluster 2024-02-10T09:40:08.933666+0000 mon.a (mon.0) 835 : cluster [INF] Cluster is now healthy 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: audit 2024-02-10T09:40:08.934652+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:40:10.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: audit 2024-02-10T09:40:08.942880+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cephadm 2024-02-10T09:40:08.947683+0000 mgr.x (mgr.34700) 11 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cephadm 2024-02-10T09:40:08.954033+0000 mgr.x (mgr.34700) 12 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[35780]: cluster 2024-02-10T09:40:09.934916+0000 mon.a (mon.0) 838 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: audit 2024-02-10T09:40:08.942880+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cephadm 2024-02-10T09:40:08.947683+0000 mgr.x (mgr.34700) 11 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cephadm 2024-02-10T09:40:08.954033+0000 mgr.x (mgr.34700) 12 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:40:10.449 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:09 smithi153 bash[39117]: cluster 2024-02-10T09:40:09.934916+0000 mon.a (mon.0) 838 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:11.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:11.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:11.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:11.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[54004]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:40:11.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:40:11.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:40:11.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:11 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:11.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: debug 2024-02-10T09:40:11.555+0000 7fc22683d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:11.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:11.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:40:11.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:11.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: main() 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:11.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: main() 2024-02-10T09:40:11.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:11.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:40:11.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:11.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:40:11.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:11.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:40:11.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:11.838 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:11.838 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:11.838 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:11.838 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:11.838 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:12.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:11 smithi204 bash[35809]: cluster 2024-02-10T09:40:10.932990+0000 mgr.x (mgr.34700) 13 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:11 smithi153 bash[35780]: cluster 2024-02-10T09:40:10.932990+0000 mgr.x (mgr.34700) 13 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:12.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:11 smithi153 bash[39117]: cluster 2024-02-10T09:40:10.932990+0000 mgr.x (mgr.34700) 13 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: cephadm 2024-02-10T09:40:11.556622+0000 mgr.x (mgr.34700) 14 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: main() 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:13.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: main() 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:40:13.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:13.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:13.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:13.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:13.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:12 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: cephadm 2024-02-10T09:40:11.556622+0000 mgr.x (mgr.34700) 14 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:40:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: main() 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:13.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: cephadm 2024-02-10T09:40:11.556622+0000 mgr.x (mgr.34700) 14 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:13.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: main() 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:13.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: main() 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:13.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: main() 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:40:13.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:40:13.447 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:12 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:40:13.966 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:13] "GET /metrics HTTP/1.1" 200 32892 "" "Prometheus/2.33.4" 2024-02-10T09:40:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:13 smithi204 bash[35809]: cluster 2024-02-10T09:40:12.933752+0000 mgr.x (mgr.34700) 15 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:14.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:13 smithi153 bash[39117]: cluster 2024-02-10T09:40:12.933752+0000 mgr.x (mgr.34700) 15 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:14.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:13 smithi153 bash[35780]: cluster 2024-02-10T09:40:12.933752+0000 mgr.x (mgr.34700) 15 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:40:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:15 smithi204 bash[35809]: cluster 2024-02-10T09:40:14.934172+0000 mgr.x (mgr.34700) 16 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:16.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:15 smithi153 bash[35780]: cluster 2024-02-10T09:40:14.934172+0000 mgr.x (mgr.34700) 16 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:16.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:15 smithi153 bash[39117]: cluster 2024-02-10T09:40:14.934172+0000 mgr.x (mgr.34700) 16 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:17 smithi153 bash[35780]: cluster 2024-02-10T09:40:16.934842+0000 mgr.x (mgr.34700) 17 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:17 smithi153 bash[39117]: cluster 2024-02-10T09:40:16.934842+0000 mgr.x (mgr.34700) 17 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:17 smithi204 bash[35809]: cluster 2024-02-10T09:40:16.934842+0000 mgr.x (mgr.34700) 17 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:19.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:19 smithi204 bash[35809]: cluster 2024-02-10T09:40:18.935325+0000 mgr.x (mgr.34700) 18 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:19 smithi153 bash[39117]: cluster 2024-02-10T09:40:18.935325+0000 mgr.x (mgr.34700) 18 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:19 smithi153 bash[35780]: cluster 2024-02-10T09:40:18.935325+0000 mgr.x (mgr.34700) 18 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:21 smithi204 bash[35809]: cluster 2024-02-10T09:40:20.935801+0000 mgr.x (mgr.34700) 19 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:21 smithi153 bash[35780]: cluster 2024-02-10T09:40:20.935801+0000 mgr.x (mgr.34700) 19 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:21.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:21 smithi153 bash[39117]: cluster 2024-02-10T09:40:20.935801+0000 mgr.x (mgr.34700) 19 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:22.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:40:22.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:40:22.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:22.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:22.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:22.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:22.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:22.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:22.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:22.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:22.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 bash[54022]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:40:22.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:40:22.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:40:22.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:21 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:23 smithi204 bash[35809]: cluster 2024-02-10T09:40:22.936778+0000 mgr.x (mgr.34700) 20 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:23.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:23 smithi153 bash[35780]: cluster 2024-02-10T09:40:22.936778+0000 mgr.x (mgr.34700) 20 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:23.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:23 smithi153 bash[39117]: cluster 2024-02-10T09:40:22.936778+0000 mgr.x (mgr.34700) 20 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:23] "GET /metrics HTTP/1.1" 200 153249 "" "Prometheus/2.33.4" 2024-02-10T09:40:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:40:25.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:25 smithi204 bash[35809]: cluster 2024-02-10T09:40:24.937249+0000 mgr.x (mgr.34700) 21 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:25 smithi153 bash[35780]: cluster 2024-02-10T09:40:24.937249+0000 mgr.x (mgr.34700) 21 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:25.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:25 smithi153 bash[39117]: cluster 2024-02-10T09:40:24.937249+0000 mgr.x (mgr.34700) 21 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:27 smithi153 bash[35780]: cluster 2024-02-10T09:40:26.938056+0000 mgr.x (mgr.34700) 22 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:27 smithi153 bash[39117]: cluster 2024-02-10T09:40:26.938056+0000 mgr.x (mgr.34700) 22 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:27 smithi204 bash[35809]: cluster 2024-02-10T09:40:26.938056+0000 mgr.x (mgr.34700) 22 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:29 smithi204 bash[35809]: cluster 2024-02-10T09:40:28.938529+0000 mgr.x (mgr.34700) 23 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:29.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:29 smithi153 bash[39117]: cluster 2024-02-10T09:40:28.938529+0000 mgr.x (mgr.34700) 23 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:29.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:29 smithi153 bash[35780]: cluster 2024-02-10T09:40:28.938529+0000 mgr.x (mgr.34700) 23 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:31 smithi204 bash[35809]: cluster 2024-02-10T09:40:30.938980+0000 mgr.x (mgr.34700) 24 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:31.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:31 smithi153 bash[35780]: cluster 2024-02-10T09:40:30.938980+0000 mgr.x (mgr.34700) 24 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:31 smithi153 bash[39117]: cluster 2024-02-10T09:40:30.938980+0000 mgr.x (mgr.34700) 24 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:32.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:32.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:32.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:32.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:32.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:32.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:32.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 bash[54039]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:40:32.335 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:40:32.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:40:32.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:31 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:33.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:33 smithi204 bash[35809]: cluster 2024-02-10T09:40:32.939842+0000 mgr.x (mgr.34700) 25 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:33.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:33 smithi153 bash[35780]: cluster 2024-02-10T09:40:32.939842+0000 mgr.x (mgr.34700) 25 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:33.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:33 smithi153 bash[39117]: cluster 2024-02-10T09:40:32.939842+0000 mgr.x (mgr.34700) 25 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:33] "GET /metrics HTTP/1.1" 200 153251 "" "Prometheus/2.33.4" 2024-02-10T09:40:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:40:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:35 smithi204 bash[35809]: cluster 2024-02-10T09:40:34.940264+0000 mgr.x (mgr.34700) 26 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:35 smithi153 bash[35780]: cluster 2024-02-10T09:40:34.940264+0000 mgr.x (mgr.34700) 26 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:35.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:35 smithi153 bash[39117]: cluster 2024-02-10T09:40:34.940264+0000 mgr.x (mgr.34700) 26 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:37 smithi153 bash[39117]: cluster 2024-02-10T09:40:36.941111+0000 mgr.x (mgr.34700) 27 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:37 smithi153 bash[35780]: cluster 2024-02-10T09:40:36.941111+0000 mgr.x (mgr.34700) 27 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:37 smithi204 bash[35809]: cluster 2024-02-10T09:40:36.941111+0000 mgr.x (mgr.34700) 27 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:39.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:39 smithi204 bash[35809]: cluster 2024-02-10T09:40:38.941632+0000 mgr.x (mgr.34700) 28 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:39 smithi153 bash[35780]: cluster 2024-02-10T09:40:38.941632+0000 mgr.x (mgr.34700) 28 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:39.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:39 smithi153 bash[39117]: cluster 2024-02-10T09:40:38.941632+0000 mgr.x (mgr.34700) 28 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:41.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:41 smithi204 bash[35809]: cluster 2024-02-10T09:40:40.942264+0000 mgr.x (mgr.34700) 29 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:41 smithi153 bash[35780]: cluster 2024-02-10T09:40:40.942264+0000 mgr.x (mgr.34700) 29 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:41.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:41 smithi153 bash[39117]: cluster 2024-02-10T09:40:40.942264+0000 mgr.x (mgr.34700) 29 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:42.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:42.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:42.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:42.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:42.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 bash[54057]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:40:42.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:40:42.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:40:42.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:42 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:43 smithi204 bash[35809]: cluster 2024-02-10T09:40:42.943006+0000 mgr.x (mgr.34700) 30 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:43.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:43 smithi153 bash[35780]: cluster 2024-02-10T09:40:42.943006+0000 mgr.x (mgr.34700) 30 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:43.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:43 smithi153 bash[39117]: cluster 2024-02-10T09:40:42.943006+0000 mgr.x (mgr.34700) 30 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:43] "GET /metrics HTTP/1.1" 200 153251 "" "Prometheus/2.33.4" 2024-02-10T09:40:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:40:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:45 smithi204 bash[35809]: cluster 2024-02-10T09:40:44.943447+0000 mgr.x (mgr.34700) 31 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:45 smithi153 bash[35780]: cluster 2024-02-10T09:40:44.943447+0000 mgr.x (mgr.34700) 31 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:45 smithi153 bash[39117]: cluster 2024-02-10T09:40:44.943447+0000 mgr.x (mgr.34700) 31 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:45.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:40:45 smithi153 bash[64342]: time="2024-02-10T09:40:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:40:47.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:40:46 smithi204 bash[52730]: time="2024-02-10T09:40:46Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:40:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:47 smithi153 bash[35780]: cluster 2024-02-10T09:40:46.944118+0000 mgr.x (mgr.34700) 32 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:47 smithi153 bash[39117]: cluster 2024-02-10T09:40:46.944118+0000 mgr.x (mgr.34700) 32 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:47 smithi204 bash[35809]: cluster 2024-02-10T09:40:46.944118+0000 mgr.x (mgr.34700) 32 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:49.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:49 smithi204 bash[35809]: cluster 2024-02-10T09:40:48.944528+0000 mgr.x (mgr.34700) 33 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:49.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:49 smithi153 bash[35780]: cluster 2024-02-10T09:40:48.944528+0000 mgr.x (mgr.34700) 33 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:49.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:49 smithi153 bash[39117]: cluster 2024-02-10T09:40:48.944528+0000 mgr.x (mgr.34700) 33 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:51.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:51 smithi204 bash[35809]: cluster 2024-02-10T09:40:50.945117+0000 mgr.x (mgr.34700) 34 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:51.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:51 smithi153 bash[35780]: cluster 2024-02-10T09:40:50.945117+0000 mgr.x (mgr.34700) 34 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:51.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:51 smithi153 bash[39117]: cluster 2024-02-10T09:40:50.945117+0000 mgr.x (mgr.34700) 34 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:40:52.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 bash[54074]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:40:52.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:40:52 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:40:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:53 smithi204 bash[35809]: cluster 2024-02-10T09:40:52.945728+0000 mgr.x (mgr.34700) 35 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:53 smithi153 bash[35780]: cluster 2024-02-10T09:40:52.945728+0000 mgr.x (mgr.34700) 35 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:53.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:53 smithi153 bash[39117]: cluster 2024-02-10T09:40:52.945728+0000 mgr.x (mgr.34700) 35 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:40:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:53] "GET /metrics HTTP/1.1" 200 153249 "" "Prometheus/2.33.4" 2024-02-10T09:40:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:40:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:40:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:40:55.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:55 smithi204 bash[35809]: cluster 2024-02-10T09:40:54.946263+0000 mgr.x (mgr.34700) 36 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:55 smithi153 bash[35780]: cluster 2024-02-10T09:40:54.946263+0000 mgr.x (mgr.34700) 36 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:55.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:55 smithi153 bash[39117]: cluster 2024-02-10T09:40:54.946263+0000 mgr.x (mgr.34700) 36 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:58.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:57 smithi153 bash[35780]: cluster 2024-02-10T09:40:56.946936+0000 mgr.x (mgr.34700) 37 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:58.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:57 smithi153 bash[39117]: cluster 2024-02-10T09:40:56.946936+0000 mgr.x (mgr.34700) 37 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:57 smithi204 bash[35809]: cluster 2024-02-10T09:40:56.946936+0000 mgr.x (mgr.34700) 37 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:59.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:40:59 smithi204 bash[35809]: cluster 2024-02-10T09:40:58.947458+0000 mgr.x (mgr.34700) 38 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:59.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:40:59 smithi153 bash[35780]: cluster 2024-02-10T09:40:58.947458+0000 mgr.x (mgr.34700) 38 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:40:59.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:40:59 smithi153 bash[39117]: cluster 2024-02-10T09:40:58.947458+0000 mgr.x (mgr.34700) 38 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.912039+0000 mon.b (mon.2) 339 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.912287+0000 mon.b (mon.2) 340 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.912454+0000 mon.b (mon.2) 341 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.912630+0000 mon.b (mon.2) 342 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.912677+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.913150+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.913348+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:00 smithi204 bash[35809]: audit 2024-02-10T09:41:00.913540+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.912039+0000 mon.b (mon.2) 339 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-10T09:41:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.912287+0000 mon.b (mon.2) 340 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-10T09:41:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.912454+0000 mon.b (mon.2) 341 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.912630+0000 mon.b (mon.2) 342 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.912677+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-10T09:41:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.913150+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-10T09:41:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.913348+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[39117]: audit 2024-02-10T09:41:00.913540+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.912039+0000 mon.b (mon.2) 339 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.912287+0000 mon.b (mon.2) 340 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.912454+0000 mon.b (mon.2) 341 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.912630+0000 mon.b (mon.2) 342 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.912677+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.913150+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.913348+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:41:01.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:00 smithi153 bash[35780]: audit 2024-02-10T09:41:00.913540+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-10T09:41:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:01 smithi204 bash[35809]: cluster 2024-02-10T09:41:00.948114+0000 mgr.x (mgr.34700) 39 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:02.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:01 smithi153 bash[35780]: cluster 2024-02-10T09:41:00.948114+0000 mgr.x (mgr.34700) 39 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:02.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:01 smithi153 bash[39117]: cluster 2024-02-10T09:41:00.948114+0000 mgr.x (mgr.34700) 39 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:03.081 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:41:03.081 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:41:03.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:02 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:03.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-02-10T09:41:03.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:41:03.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:02 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:03.979 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:03] "GET /metrics HTTP/1.1" 200 153245 "" "Prometheus/2.33.4" 2024-02-10T09:41:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:03 smithi204 bash[35809]: cluster 2024-02-10T09:41:02.948982+0000 mgr.x (mgr.34700) 40 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:04.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:03 smithi153 bash[35780]: cluster 2024-02-10T09:41:02.948982+0000 mgr.x (mgr.34700) 40 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:03 smithi153 bash[39117]: cluster 2024-02-10T09:41:02.948982+0000 mgr.x (mgr.34700) 40 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:41:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:41:06.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:05 smithi204 bash[35809]: cluster 2024-02-10T09:41:04.949412+0000 mgr.x (mgr.34700) 41 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:06.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:05 smithi153 bash[35780]: cluster 2024-02-10T09:41:04.949412+0000 mgr.x (mgr.34700) 41 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:06.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:05 smithi153 bash[39117]: cluster 2024-02-10T09:41:04.949412+0000 mgr.x (mgr.34700) 41 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:07 smithi153 bash[35780]: cluster 2024-02-10T09:41:06.950342+0000 mgr.x (mgr.34700) 42 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:07 smithi153 bash[39117]: cluster 2024-02-10T09:41:06.950342+0000 mgr.x (mgr.34700) 42 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:07 smithi204 bash[35809]: cluster 2024-02-10T09:41:06.950342+0000 mgr.x (mgr.34700) 42 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:09.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:09 smithi204 bash[35809]: cluster 2024-02-10T09:41:08.950777+0000 mgr.x (mgr.34700) 43 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:09 smithi153 bash[35780]: cluster 2024-02-10T09:41:08.950777+0000 mgr.x (mgr.34700) 43 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:09 smithi153 bash[39117]: cluster 2024-02-10T09:41:08.950777+0000 mgr.x (mgr.34700) 43 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:11 smithi204 bash[35809]: cluster 2024-02-10T09:41:10.951357+0000 mgr.x (mgr.34700) 44 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:11 smithi153 bash[35780]: cluster 2024-02-10T09:41:10.951357+0000 mgr.x (mgr.34700) 44 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:11 smithi153 bash[39117]: cluster 2024-02-10T09:41:10.951357+0000 mgr.x (mgr.34700) 44 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: audit 2024-02-10T09:41:11.557915+0000 mon.b (mon.2) 343 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: cluster 2024-02-10T09:41:11.858468+0000 mgr.x (mgr.34700) 45 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: cluster 2024-02-10T09:41:11.858659+0000 mgr.x (mgr.34700) 46 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: audit 2024-02-10T09:41:11.859137+0000 mon.b (mon.2) 344 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: audit 2024-02-10T09:41:11.859420+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: audit 2024-02-10T09:41:11.864981+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: cephadm 2024-02-10T09:41:11.868627+0000 mgr.x (mgr.34700) 47 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:41:12.326 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:12 smithi204 bash[35809]: cephadm 2024-02-10T09:41:11.874440+0000 mgr.x (mgr.34700) 48 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:41:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: audit 2024-02-10T09:41:11.557915+0000 mon.b (mon.2) 343 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:41:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: cluster 2024-02-10T09:41:11.858468+0000 mgr.x (mgr.34700) 45 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: cluster 2024-02-10T09:41:11.858659+0000 mgr.x (mgr.34700) 46 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: audit 2024-02-10T09:41:11.859137+0000 mon.b (mon.2) 344 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:41:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: audit 2024-02-10T09:41:11.859420+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:41:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: audit 2024-02-10T09:41:11.864981+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: cephadm 2024-02-10T09:41:11.868627+0000 mgr.x (mgr.34700) 47 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[35780]: cephadm 2024-02-10T09:41:11.874440+0000 mgr.x (mgr.34700) 48 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: audit 2024-02-10T09:41:11.557915+0000 mon.b (mon.2) 343 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: cluster 2024-02-10T09:41:11.858468+0000 mgr.x (mgr.34700) 45 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: cluster 2024-02-10T09:41:11.858659+0000 mgr.x (mgr.34700) 46 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: audit 2024-02-10T09:41:11.859137+0000 mon.b (mon.2) 344 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: audit 2024-02-10T09:41:11.859420+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: audit 2024-02-10T09:41:11.864981+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: cephadm 2024-02-10T09:41:11.868627+0000 mgr.x (mgr.34700) 47 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:41:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:12 smithi153 bash[39117]: cephadm 2024-02-10T09:41:11.874440+0000 mgr.x (mgr.34700) 48 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:41:13.631 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:13] "GET /metrics HTTP/1.1" 200 153245 "" "Prometheus/2.33.4" 2024-02-10T09:41:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:41:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:41:14.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[54316]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:41:14.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:41:14.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:41:14.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:14 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: debug 2024-02-10T09:41:14.637+0000 7fc22683d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:14.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: main() 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:14.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: main() 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:14.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:14.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:14 smithi153 bash[35780]: cluster 2024-02-10T09:41:13.858994+0000 mgr.x (mgr.34700) 49 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:14 smithi153 bash[39117]: cluster 2024-02-10T09:41:13.858994+0000 mgr.x (mgr.34700) 49 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:14 smithi204 bash[35809]: cluster 2024-02-10T09:41:13.858994+0000 mgr.x (mgr.34700) 49 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: cephadm 2024-02-10T09:41:14.638763+0000 mgr.x (mgr.34700) 50 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:41:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: main() 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:16.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:41:16.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: cephadm 2024-02-10T09:41:14.638763+0000 mgr.x (mgr.34700) 50 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: main() 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:16.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:41:16.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: main() 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:41:16.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: main() 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:16.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:15 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: cephadm 2024-02-10T09:41:14.638763+0000 mgr.x (mgr.34700) 50 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:16.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: main() 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:41:16.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: main() 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:41:16.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:41:16.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:15 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:41:17.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:16 smithi153 bash[35780]: cluster 2024-02-10T09:41:15.859421+0000 mgr.x (mgr.34700) 51 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:17.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:16 smithi153 bash[39117]: cluster 2024-02-10T09:41:15.859421+0000 mgr.x (mgr.34700) 51 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:17.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:16 smithi204 bash[35809]: cluster 2024-02-10T09:41:15.859421+0000 mgr.x (mgr.34700) 51 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:19.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:18 smithi153 bash[35780]: cluster 2024-02-10T09:41:17.860295+0000 mgr.x (mgr.34700) 52 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:18 smithi153 bash[39117]: cluster 2024-02-10T09:41:17.860295+0000 mgr.x (mgr.34700) 52 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:19.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:18 smithi204 bash[35809]: cluster 2024-02-10T09:41:17.860295+0000 mgr.x (mgr.34700) 52 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:21.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:20 smithi153 bash[35780]: cluster 2024-02-10T09:41:19.860812+0000 mgr.x (mgr.34700) 53 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:21.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:20 smithi153 bash[39117]: cluster 2024-02-10T09:41:19.860812+0000 mgr.x (mgr.34700) 53 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:20 smithi204 bash[35809]: cluster 2024-02-10T09:41:19.860812+0000 mgr.x (mgr.34700) 53 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:23.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:22 smithi153 bash[39117]: cluster 2024-02-10T09:41:21.861509+0000 mgr.x (mgr.34700) 54 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:23.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:22 smithi153 bash[35780]: cluster 2024-02-10T09:41:21.861509+0000 mgr.x (mgr.34700) 54 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:23.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:41:23 smithi153 bash[64342]: time="2024-02-10T09:41:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:41:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:22 smithi204 bash[35809]: cluster 2024-02-10T09:41:21.861509+0000 mgr.x (mgr.34700) 54 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:23] "GET /metrics HTTP/1.1" 200 153248 "" "Prometheus/2.33.4" 2024-02-10T09:41:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:41:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:41:24.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:41:24 smithi204 bash[52730]: time="2024-02-10T09:41:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:41:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:24 smithi153 bash[35780]: cluster 2024-02-10T09:41:23.862072+0000 mgr.x (mgr.34700) 55 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:25.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:24 smithi153 bash[39117]: cluster 2024-02-10T09:41:23.862072+0000 mgr.x (mgr.34700) 55 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:25.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:24 smithi204 bash[35809]: cluster 2024-02-10T09:41:23.862072+0000 mgr.x (mgr.34700) 55 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:25.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 bash[54333]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:41:25.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:24 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:27.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:26 smithi153 bash[35780]: cluster 2024-02-10T09:41:25.862652+0000 mgr.x (mgr.34700) 56 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:26 smithi153 bash[39117]: cluster 2024-02-10T09:41:25.862652+0000 mgr.x (mgr.34700) 56 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:27.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:26 smithi204 bash[35809]: cluster 2024-02-10T09:41:25.862652+0000 mgr.x (mgr.34700) 56 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:29.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:28 smithi153 bash[35780]: cluster 2024-02-10T09:41:27.863420+0000 mgr.x (mgr.34700) 57 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:28 smithi153 bash[39117]: cluster 2024-02-10T09:41:27.863420+0000 mgr.x (mgr.34700) 57 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:29.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:28 smithi204 bash[35809]: cluster 2024-02-10T09:41:27.863420+0000 mgr.x (mgr.34700) 57 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:31.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:30 smithi153 bash[35780]: cluster 2024-02-10T09:41:29.864005+0000 mgr.x (mgr.34700) 58 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:31.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:30 smithi153 bash[39117]: cluster 2024-02-10T09:41:29.864005+0000 mgr.x (mgr.34700) 58 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:30 smithi204 bash[35809]: cluster 2024-02-10T09:41:29.864005+0000 mgr.x (mgr.34700) 58 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:33.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:32 smithi153 bash[35780]: cluster 2024-02-10T09:41:31.864775+0000 mgr.x (mgr.34700) 59 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:33.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:32 smithi153 bash[39117]: cluster 2024-02-10T09:41:31.864775+0000 mgr.x (mgr.34700) 59 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:33.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:32 smithi204 bash[35809]: cluster 2024-02-10T09:41:31.864775+0000 mgr.x (mgr.34700) 59 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:33] "GET /metrics HTTP/1.1" 200 153244 "" "Prometheus/2.33.4" 2024-02-10T09:41:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:41:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:41:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:34 smithi153 bash[35780]: cluster 2024-02-10T09:41:33.865198+0000 mgr.x (mgr.34700) 60 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:34 smithi153 bash[39117]: cluster 2024-02-10T09:41:33.865198+0000 mgr.x (mgr.34700) 60 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:35.199 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:34 smithi204 bash[35809]: cluster 2024-02-10T09:41:33.865198+0000 mgr.x (mgr.34700) 60 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:41:35.200 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:35.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:35.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:35.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:35.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:35.201 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:35.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 bash[54351]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:41:35.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:41:35.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:41:35.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:35 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:37.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:36 smithi153 bash[35780]: cluster 2024-02-10T09:41:35.865687+0000 mgr.x (mgr.34700) 61 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:37.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:36 smithi153 bash[39117]: cluster 2024-02-10T09:41:35.865687+0000 mgr.x (mgr.34700) 61 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:36 smithi204 bash[35809]: cluster 2024-02-10T09:41:35.865687+0000 mgr.x (mgr.34700) 61 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:39.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:38 smithi153 bash[35780]: cluster 2024-02-10T09:41:37.866503+0000 mgr.x (mgr.34700) 62 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:39.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:38 smithi153 bash[39117]: cluster 2024-02-10T09:41:37.866503+0000 mgr.x (mgr.34700) 62 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:39.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:38 smithi204 bash[35809]: cluster 2024-02-10T09:41:37.866503+0000 mgr.x (mgr.34700) 62 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:41.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:40 smithi153 bash[35780]: cluster 2024-02-10T09:41:39.866963+0000 mgr.x (mgr.34700) 63 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:41.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:40 smithi153 bash[39117]: cluster 2024-02-10T09:41:39.866963+0000 mgr.x (mgr.34700) 63 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:41.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:40 smithi204 bash[35809]: cluster 2024-02-10T09:41:39.866963+0000 mgr.x (mgr.34700) 63 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:43.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:42 smithi153 bash[39117]: cluster 2024-02-10T09:41:41.867684+0000 mgr.x (mgr.34700) 64 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:42 smithi153 bash[35780]: cluster 2024-02-10T09:41:41.867684+0000 mgr.x (mgr.34700) 64 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:43.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:42 smithi204 bash[35809]: cluster 2024-02-10T09:41:41.867684+0000 mgr.x (mgr.34700) 64 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:43] "GET /metrics HTTP/1.1" 200 153244 "" "Prometheus/2.33.4" 2024-02-10T09:41:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:41:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:41:45.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:44 smithi153 bash[35780]: cluster 2024-02-10T09:41:43.868114+0000 mgr.x (mgr.34700) 65 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:44 smithi153 bash[39117]: cluster 2024-02-10T09:41:43.868114+0000 mgr.x (mgr.34700) 65 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:44 smithi204 bash[35809]: cluster 2024-02-10T09:41:43.868114+0000 mgr.x (mgr.34700) 65 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:45.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:45.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:45.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:45.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:45.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 bash[54368]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:41:45.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:41:45.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:41:45.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:45 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:47.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:46 smithi153 bash[39117]: cluster 2024-02-10T09:41:45.868713+0000 mgr.x (mgr.34700) 66 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:47.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:46 smithi153 bash[35780]: cluster 2024-02-10T09:41:45.868713+0000 mgr.x (mgr.34700) 66 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:47.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:46 smithi204 bash[35809]: cluster 2024-02-10T09:41:45.868713+0000 mgr.x (mgr.34700) 66 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:49.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:48 smithi153 bash[35780]: cluster 2024-02-10T09:41:47.869422+0000 mgr.x (mgr.34700) 67 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:49.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:48 smithi153 bash[39117]: cluster 2024-02-10T09:41:47.869422+0000 mgr.x (mgr.34700) 67 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:49.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:48 smithi204 bash[35809]: cluster 2024-02-10T09:41:47.869422+0000 mgr.x (mgr.34700) 67 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:50 smithi153 bash[35780]: cluster 2024-02-10T09:41:49.870076+0000 mgr.x (mgr.34700) 68 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:51.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:50 smithi153 bash[39117]: cluster 2024-02-10T09:41:49.870076+0000 mgr.x (mgr.34700) 68 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:51.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:50 smithi204 bash[35809]: cluster 2024-02-10T09:41:49.870076+0000 mgr.x (mgr.34700) 68 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:53.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:52 smithi153 bash[35780]: cluster 2024-02-10T09:41:51.870816+0000 mgr.x (mgr.34700) 69 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:53.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:52 smithi153 bash[39117]: cluster 2024-02-10T09:41:51.870816+0000 mgr.x (mgr.34700) 69 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:52 smithi204 bash[35809]: cluster 2024-02-10T09:41:51.870816+0000 mgr.x (mgr.34700) 69 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:41:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:53] "GET /metrics HTTP/1.1" 200 153249 "" "Prometheus/2.33.4" 2024-02-10T09:41:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:41:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:41:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:41:55.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:54 smithi153 bash[35780]: cluster 2024-02-10T09:41:53.871396+0000 mgr.x (mgr.34700) 70 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:55.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:54 smithi153 bash[39117]: cluster 2024-02-10T09:41:53.871396+0000 mgr.x (mgr.34700) 70 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:55.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:54 smithi204 bash[35809]: cluster 2024-02-10T09:41:53.871396+0000 mgr.x (mgr.34700) 70 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:41:56.082 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 bash[54386]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:41:56.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:41:55 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:41:57.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:56 smithi153 bash[39117]: cluster 2024-02-10T09:41:55.871801+0000 mgr.x (mgr.34700) 71 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:57.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:56 smithi153 bash[35780]: cluster 2024-02-10T09:41:55.871801+0000 mgr.x (mgr.34700) 71 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:57.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:56 smithi204 bash[35809]: cluster 2024-02-10T09:41:55.871801+0000 mgr.x (mgr.34700) 71 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:59.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:41:58 smithi153 bash[39117]: cluster 2024-02-10T09:41:57.872577+0000 mgr.x (mgr.34700) 72 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:59.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:41:58 smithi153 bash[35780]: cluster 2024-02-10T09:41:57.872577+0000 mgr.x (mgr.34700) 72 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:41:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:41:58 smithi204 bash[35809]: cluster 2024-02-10T09:41:57.872577+0000 mgr.x (mgr.34700) 72 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:01.081 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:00 smithi153 bash[39117]: cluster 2024-02-10T09:41:59.873108+0000 mgr.x (mgr.34700) 73 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:01.082 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:00 smithi153 bash[35780]: cluster 2024-02-10T09:41:59.873108+0000 mgr.x (mgr.34700) 73 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:01.082 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:00 smithi153 bash[64342]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:42:01.082 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:00 smithi153 bash[64342]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:42:01.082 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[64342]: Error: 3 errors occurred while pulling: 2024-02-10T09:42:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:00 smithi204 bash[35809]: cluster 2024-02-10T09:41:59.873108+0000 mgr.x (mgr.34700) 73 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:01.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[64342]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:42:01.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[64342]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:42:01.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[64342]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:42:01.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Control process exited, code=exited status=125 2024-02-10T09:42:01.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:01.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:01 smithi153 systemd[1]: Failed to start Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:02.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[52730]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:42:02.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[52730]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:42:02.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.916752+0000 mon.b (mon.2) 345 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-02-10T09:42:02.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.916978+0000 mon.b (mon.2) 346 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.917163+0000 mon.b (mon.2) 347 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.917271+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-02-10T09:42:02.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.917346+0000 mon.b (mon.2) 348 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T09:42:02.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.917620+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.917871+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:01 smithi204 bash[35809]: audit 2024-02-10T09:42:00.918038+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.916752+0000 mon.b (mon.2) 345 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.916978+0000 mon.b (mon.2) 346 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.917163+0000 mon.b (mon.2) 347 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.917271+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.917346+0000 mon.b (mon.2) 348 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.917620+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.917871+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[35780]: audit 2024-02-10T09:42:00.918038+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.916752+0000 mon.b (mon.2) 345 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.916978+0000 mon.b (mon.2) 346 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.917163+0000 mon.b (mon.2) 347 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.917271+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.917346+0000 mon.b (mon.2) 348 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.917620+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.917871+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:42:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:01 smithi153 bash[39117]: audit 2024-02-10T09:42:00.918038+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 bash[52730]: Error: 3 errors occurred while pulling: 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 bash[52730]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 bash[52730]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 bash[52730]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Control process exited, code=exited status=125 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:42:02.582 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:02 smithi204 systemd[1]: Failed to start Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:03.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:02 smithi153 bash[35780]: cluster 2024-02-10T09:42:01.873882+0000 mgr.x (mgr.34700) 74 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:02 smithi153 bash[39117]: cluster 2024-02-10T09:42:01.873882+0000 mgr.x (mgr.34700) 74 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:02 smithi204 bash[35809]: cluster 2024-02-10T09:42:01.873882+0000 mgr.x (mgr.34700) 74 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:03] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:42:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:42:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:42:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:04 smithi153 bash[35780]: cluster 2024-02-10T09:42:03.874518+0000 mgr.x (mgr.34700) 75 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:04 smithi153 bash[39117]: cluster 2024-02-10T09:42:03.874518+0000 mgr.x (mgr.34700) 75 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:04 smithi204 bash[35809]: cluster 2024-02-10T09:42:03.874518+0000 mgr.x (mgr.34700) 75 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:06.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:05 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:06.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:05 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:42:06.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:05 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:06.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:05 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-02-10T09:42:06.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:05 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:06.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:05 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:06 smithi153 bash[35780]: cluster 2024-02-10T09:42:05.875055+0000 mgr.x (mgr.34700) 76 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:06 smithi153 bash[39117]: cluster 2024-02-10T09:42:05.875055+0000 mgr.x (mgr.34700) 76 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:07.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:06 smithi204 bash[35809]: cluster 2024-02-10T09:42:05.875055+0000 mgr.x (mgr.34700) 76 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:08 smithi153 bash[35780]: cluster 2024-02-10T09:42:07.875757+0000 mgr.x (mgr.34700) 77 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:08 smithi153 bash[39117]: cluster 2024-02-10T09:42:07.875757+0000 mgr.x (mgr.34700) 77 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:09.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:08 smithi204 bash[35809]: cluster 2024-02-10T09:42:07.875757+0000 mgr.x (mgr.34700) 77 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:11.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:10 smithi153 bash[39117]: cluster 2024-02-10T09:42:09.876268+0000 mgr.x (mgr.34700) 78 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:10 smithi153 bash[35780]: cluster 2024-02-10T09:42:09.876268+0000 mgr.x (mgr.34700) 78 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:11.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:10 smithi204 bash[35809]: cluster 2024-02-10T09:42:09.876268+0000 mgr.x (mgr.34700) 78 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:11.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:11 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:11.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:11 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Scheduled restart job, restart counter is at 6. 2024-02-10T09:42:11.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:11 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:11.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:11 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:11.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:11 smithi153 bash[66122]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:42:11.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:11 smithi153 bash[66122]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:42:12.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:12.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:12 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Scheduled restart job, restart counter is at 6. 2024-02-10T09:42:12.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:12 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:12.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:12 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:12.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:12 smithi204 bash[54417]: Resolving "prom/node-exporter" using unqualified-search registries (/etc/containers/registries.conf) 2024-02-10T09:42:12.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:12 smithi204 bash[54417]: Trying to pull docker.io/prom/node-exporter:latest... 2024-02-10T09:42:13.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:12 smithi153 bash[35780]: cluster 2024-02-10T09:42:11.876971+0000 mgr.x (mgr.34700) 79 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:13.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:12 smithi153 bash[39117]: cluster 2024-02-10T09:42:11.876971+0000 mgr.x (mgr.34700) 79 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:13.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:12 smithi204 bash[35809]: cluster 2024-02-10T09:42:11.876971+0000 mgr.x (mgr.34700) 79 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:13 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:13] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:42:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:42:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:42:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:14 smithi153 bash[35780]: cluster 2024-02-10T09:42:13.877428+0000 mgr.x (mgr.34700) 80 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:14 smithi153 bash[35780]: audit 2024-02-10T09:42:14.640490+0000 mon.b (mon.2) 349 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:42:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:14 smithi153 bash[39117]: cluster 2024-02-10T09:42:13.877428+0000 mgr.x (mgr.34700) 80 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:14 smithi153 bash[39117]: audit 2024-02-10T09:42:14.640490+0000 mon.b (mon.2) 349 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:42:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:14 smithi204 bash[35809]: cluster 2024-02-10T09:42:13.877428+0000 mgr.x (mgr.34700) 80 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:15.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:14 smithi204 bash[35809]: audit 2024-02-10T09:42:14.640490+0000 mon.b (mon.2) 349 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: cluster 2024-02-10T09:42:14.937472+0000 mgr.x (mgr.34700) 81 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: cluster 2024-02-10T09:42:14.937619+0000 mgr.x (mgr.34700) 82 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: audit 2024-02-10T09:42:14.938165+0000 mon.b (mon.2) 350 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: audit 2024-02-10T09:42:14.939303+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: audit 2024-02-10T09:42:14.943009+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: cephadm 2024-02-10T09:42:14.948285+0000 mgr.x (mgr.34700) 83 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:42:16.142 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:15 smithi204 bash[35809]: cephadm 2024-02-10T09:42:14.954166+0000 mgr.x (mgr.34700) 84 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: cluster 2024-02-10T09:42:14.937472+0000 mgr.x (mgr.34700) 81 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: cluster 2024-02-10T09:42:14.937619+0000 mgr.x (mgr.34700) 82 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: audit 2024-02-10T09:42:14.938165+0000 mon.b (mon.2) 350 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: audit 2024-02-10T09:42:14.939303+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: audit 2024-02-10T09:42:14.943009+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: cephadm 2024-02-10T09:42:14.948285+0000 mgr.x (mgr.34700) 83 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:42:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[39117]: cephadm 2024-02-10T09:42:14.954166+0000 mgr.x (mgr.34700) 84 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: cluster 2024-02-10T09:42:14.937472+0000 mgr.x (mgr.34700) 81 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: cluster 2024-02-10T09:42:14.937619+0000 mgr.x (mgr.34700) 82 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: audit 2024-02-10T09:42:14.938165+0000 mon.b (mon.2) 350 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: audit 2024-02-10T09:42:14.939303+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: audit 2024-02-10T09:42:14.943009+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: cephadm 2024-02-10T09:42:14.948285+0000 mgr.x (mgr.34700) 83 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:42:16.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:15 smithi153 bash[35780]: cephadm 2024-02-10T09:42:14.954166+0000 mgr.x (mgr.34700) 84 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[35809]: cluster 2024-02-10T09:42:16.938191+0000 mgr.x (mgr.34700) 85 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: debug 2024-02-10T09:42:17.893+0000 7fc22683d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:18.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: main() 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:18.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:18.084 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:42:18.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:18.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:18.084 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[54653]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:18.085 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:17 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: main() 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:18.086 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:18.087 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:18.087 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:18.087 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:17 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:17 smithi153 bash[35780]: cluster 2024-02-10T09:42:16.938191+0000 mgr.x (mgr.34700) 85 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:18.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:17 smithi153 bash[39117]: cluster 2024-02-10T09:42:16.938191+0000 mgr.x (mgr.34700) 85 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:19.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: cephadm 2024-02-10T09:42:17.894624+0000 mgr.x (mgr.34700) 86 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: main() 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: cephadm 2024-02-10T09:42:17.894624+0000 mgr.x (mgr.34700) 86 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: main() 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:42:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: main() 2024-02-10T09:42:19.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: main() 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:18 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: cephadm 2024-02-10T09:42:17.894624+0000 mgr.x (mgr.34700) 86 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:42:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:42:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: main() 2024-02-10T09:42:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:42:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: main() 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:42:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:18 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:42:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:19 smithi153 bash[35780]: cluster 2024-02-10T09:42:18.938796+0000 mgr.x (mgr.34700) 87 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:20.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:19 smithi153 bash[39117]: cluster 2024-02-10T09:42:18.938796+0000 mgr.x (mgr.34700) 87 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:19 smithi204 bash[35809]: cluster 2024-02-10T09:42:18.938796+0000 mgr.x (mgr.34700) 87 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:22.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:21 smithi204 bash[35809]: cluster 2024-02-10T09:42:20.939336+0000 mgr.x (mgr.34700) 88 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:21 smithi153 bash[35780]: cluster 2024-02-10T09:42:20.939336+0000 mgr.x (mgr.34700) 88 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:22.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:21 smithi153 bash[39117]: cluster 2024-02-10T09:42:20.939336+0000 mgr.x (mgr.34700) 88 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:24.000 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:23] "GET /metrics HTTP/1.1" 200 153261 "" "Prometheus/2.33.4" 2024-02-10T09:42:24.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:24 smithi204 bash[35809]: cluster 2024-02-10T09:42:22.940233+0000 mgr.x (mgr.34700) 89 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:24.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:23 smithi153 bash[39117]: cluster 2024-02-10T09:42:22.940233+0000 mgr.x (mgr.34700) 89 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:24.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:23 smithi153 bash[35780]: cluster 2024-02-10T09:42:22.940233+0000 mgr.x (mgr.34700) 89 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:42:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:42:25.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:25 smithi204 bash[35809]: cluster 2024-02-10T09:42:24.940680+0000 mgr.x (mgr.34700) 90 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:25 smithi153 bash[35780]: cluster 2024-02-10T09:42:24.940680+0000 mgr.x (mgr.34700) 90 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:25.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:25 smithi153 bash[39117]: cluster 2024-02-10T09:42:24.940680+0000 mgr.x (mgr.34700) 90 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:27 smithi153 bash[35780]: cluster 2024-02-10T09:42:26.941518+0000 mgr.x (mgr.34700) 91 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:27 smithi153 bash[39117]: cluster 2024-02-10T09:42:26.941518+0000 mgr.x (mgr.34700) 91 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:28.204 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:27 smithi204 bash[35809]: cluster 2024-02-10T09:42:26.941518+0000 mgr.x (mgr.34700) 91 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:28.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:28.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:42:28.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:28.204 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:28.205 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:28.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 bash[54670]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:42:28.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:42:28.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:28.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:28 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:29 smithi204 bash[35809]: cluster 2024-02-10T09:42:28.941998+0000 mgr.x (mgr.34700) 92 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:30.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:29 smithi153 bash[35780]: cluster 2024-02-10T09:42:28.941998+0000 mgr.x (mgr.34700) 92 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:30.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:29 smithi153 bash[39117]: cluster 2024-02-10T09:42:28.941998+0000 mgr.x (mgr.34700) 92 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:31 smithi204 bash[35809]: cluster 2024-02-10T09:42:30.942589+0000 mgr.x (mgr.34700) 93 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:31.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:31 smithi153 bash[35780]: cluster 2024-02-10T09:42:30.942589+0000 mgr.x (mgr.34700) 93 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:31.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:31 smithi153 bash[39117]: cluster 2024-02-10T09:42:30.942589+0000 mgr.x (mgr.34700) 93 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:34.000 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:33] "GET /metrics HTTP/1.1" 200 153262 "" "Prometheus/2.33.4" 2024-02-10T09:42:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:33 smithi204 bash[35809]: cluster 2024-02-10T09:42:32.943356+0000 mgr.x (mgr.34700) 94 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:34.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:34 smithi153 bash[39117]: cluster 2024-02-10T09:42:32.943356+0000 mgr.x (mgr.34700) 94 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:34.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:33 smithi153 bash[35780]: cluster 2024-02-10T09:42:32.943356+0000 mgr.x (mgr.34700) 94 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:42:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:42:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:35 smithi204 bash[35809]: cluster 2024-02-10T09:42:34.943837+0000 mgr.x (mgr.34700) 95 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:35 smithi153 bash[35780]: cluster 2024-02-10T09:42:34.943837+0000 mgr.x (mgr.34700) 95 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:35.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:35 smithi153 bash[39117]: cluster 2024-02-10T09:42:34.943837+0000 mgr.x (mgr.34700) 95 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:38.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:37 smithi153 bash[35780]: cluster 2024-02-10T09:42:36.944548+0000 mgr.x (mgr.34700) 96 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:38.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:37 smithi153 bash[39117]: cluster 2024-02-10T09:42:36.944548+0000 mgr.x (mgr.34700) 96 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:37 smithi204 bash[35809]: cluster 2024-02-10T09:42:36.944548+0000 mgr.x (mgr.34700) 96 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:38.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:38.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:38.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:38.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:38.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 bash[54687]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:42:38.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:42:38.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:38.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:38 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:40 smithi204 bash[35809]: cluster 2024-02-10T09:42:38.945098+0000 mgr.x (mgr.34700) 97 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:40.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:40 smithi153 bash[35780]: cluster 2024-02-10T09:42:38.945098+0000 mgr.x (mgr.34700) 97 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:40.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:40 smithi153 bash[39117]: cluster 2024-02-10T09:42:38.945098+0000 mgr.x (mgr.34700) 97 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:41.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:41 smithi204 bash[35809]: cluster 2024-02-10T09:42:40.945625+0000 mgr.x (mgr.34700) 98 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:41 smithi153 bash[35780]: cluster 2024-02-10T09:42:40.945625+0000 mgr.x (mgr.34700) 98 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:41 smithi153 bash[39117]: cluster 2024-02-10T09:42:40.945625+0000 mgr.x (mgr.34700) 98 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:44.003 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:43 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:43] "GET /metrics HTTP/1.1" 200 153262 "" "Prometheus/2.33.4" 2024-02-10T09:42:44.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:44 smithi204 bash[35809]: cluster 2024-02-10T09:42:42.946483+0000 mgr.x (mgr.34700) 99 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:44.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:44 smithi153 bash[39117]: cluster 2024-02-10T09:42:42.946483+0000 mgr.x (mgr.34700) 99 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:44.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:44 smithi153 bash[35780]: cluster 2024-02-10T09:42:42.946483+0000 mgr.x (mgr.34700) 99 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:42:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:42:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:45 smithi204 bash[35809]: cluster 2024-02-10T09:42:44.946872+0000 mgr.x (mgr.34700) 100 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:45 smithi153 bash[35780]: cluster 2024-02-10T09:42:44.946872+0000 mgr.x (mgr.34700) 100 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:45 smithi153 bash[39117]: cluster 2024-02-10T09:42:44.946872+0000 mgr.x (mgr.34700) 100 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:47 smithi153 bash[35780]: cluster 2024-02-10T09:42:46.947713+0000 mgr.x (mgr.34700) 101 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:48.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:47 smithi153 bash[39117]: cluster 2024-02-10T09:42:46.947713+0000 mgr.x (mgr.34700) 101 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:48.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:42:47 smithi153 bash[66122]: time="2024-02-10T09:42:47Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:42:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:47 smithi204 bash[35809]: cluster 2024-02-10T09:42:46.947713+0000 mgr.x (mgr.34700) 101 : cluster [DBG] pgmap v91: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:48.867 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:48.867 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:48.868 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 bash[54705]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:42:48.869 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:42:48.869 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:48.869 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:48 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:49.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:42:48 smithi204 bash[54417]: time="2024-02-10T09:42:48Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:42:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:50 smithi204 bash[35809]: cluster 2024-02-10T09:42:48.948106+0000 mgr.x (mgr.34700) 102 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:50.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:50 smithi153 bash[35780]: cluster 2024-02-10T09:42:48.948106+0000 mgr.x (mgr.34700) 102 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:50.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:50 smithi153 bash[39117]: cluster 2024-02-10T09:42:48.948106+0000 mgr.x (mgr.34700) 102 : cluster [DBG] pgmap v92: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:51.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:51 smithi204 bash[35809]: cluster 2024-02-10T09:42:50.948703+0000 mgr.x (mgr.34700) 103 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:51.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:51 smithi153 bash[35780]: cluster 2024-02-10T09:42:50.948703+0000 mgr.x (mgr.34700) 103 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:51.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:51 smithi153 bash[39117]: cluster 2024-02-10T09:42:50.948703+0000 mgr.x (mgr.34700) 103 : cluster [DBG] pgmap v93: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:53 smithi204 bash[35809]: cluster 2024-02-10T09:42:52.949620+0000 mgr.x (mgr.34700) 104 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:53.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:53 smithi153 bash[39117]: cluster 2024-02-10T09:42:52.949620+0000 mgr.x (mgr.34700) 104 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:53 smithi153 bash[35780]: cluster 2024-02-10T09:42:52.949620+0000 mgr.x (mgr.34700) 104 : cluster [DBG] pgmap v94: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:42:53 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:53] "GET /metrics HTTP/1.1" 200 153270 "" "Prometheus/2.33.4" 2024-02-10T09:42:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:42:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:42:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:42:55.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:55 smithi204 bash[35809]: cluster 2024-02-10T09:42:54.950059+0000 mgr.x (mgr.34700) 105 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:55 smithi153 bash[35780]: cluster 2024-02-10T09:42:54.950059+0000 mgr.x (mgr.34700) 105 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:55.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:55 smithi153 bash[39117]: cluster 2024-02-10T09:42:54.950059+0000 mgr.x (mgr.34700) 105 : cluster [DBG] pgmap v95: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:57.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:42:58.074 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:57 smithi153 bash[39117]: cluster 2024-02-10T09:42:56.950812+0000 mgr.x (mgr.34700) 106 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:58.075 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:57 smithi153 bash[35780]: cluster 2024-02-10T09:42:56.950812+0000 mgr.x (mgr.34700) 106 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:57 smithi204 bash[35809]: cluster 2024-02-10T09:42:56.950812+0000 mgr.x (mgr.34700) 106 : cluster [DBG] pgmap v96: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:59.166 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:42:59 smithi153 bash[35780]: cluster 2024-02-10T09:42:58.951171+0000 mgr.x (mgr.34700) 107 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:59.166 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:42:59 smithi153 bash[39117]: cluster 2024-02-10T09:42:58.951171+0000 mgr.x (mgr.34700) 107 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:42:59 smithi204 bash[35809]: cluster 2024-02-10T09:42:58.951171+0000 mgr.x (mgr.34700) 107 : cluster [DBG] pgmap v97: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:59.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 bash[54722]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:42:59.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:42:58 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:42:59.614 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (10m) 2m ago 11m 25.9M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 error 2m ago 13m 35.2M - 6.6.2 87a51ecf0b1c 72f83803a230 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (16m) 2m ago 24m 412M - 16.2.14-521-g0e714d9a 1fbc92c859dc 46c6073a79be 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (7m) 2m ago 26m 382M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (26m) 2m ago 26m 173M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (24m) 2m ago 24m 148M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (24m) 2m ago 24m 146M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 error 2m ago 16m - - 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 error 2m ago 16m - - 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (23m) 2m ago 24m 43.4M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (23m) 2m ago 23m 44.2M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (23m) 2m ago 23m 42.7M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (23m) 2m ago 23m 45.7M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (22m) 2m ago 22m 47.5M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:42:59.615 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (22m) 2m ago 22m 48.3M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:42:59.616 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (22m) 2m ago 22m 46.2M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:42:59.616 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (21m) 2m ago 21m 45.4M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:42:59.616 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (11m) 2m ago 11m 63.8M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:42:59.616 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (13m) 2m ago 13m 34.8M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:42:59.616 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (13m) 2m ago 13m 36.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:43:00.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.602448+0000 mgr.x (mgr.34700) 108 : audit [DBG] from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.604515+0000 mon.b (mon.2) 351 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.604887+0000 mon.b (mon.2) 352 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.605203+0000 mon.b (mon.2) 353 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.605505+0000 mon.b (mon.2) 354 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.605792+0000 mon.b (mon.2) 355 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.606072+0000 mon.b (mon.2) 356 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.607875+0000 mon.b (mon.2) 357 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.608194+0000 mon.b (mon.2) 358 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.608490+0000 mon.b (mon.2) 359 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.262 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.608770+0000 mon.b (mon.2) 360 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[35780]: audit 2024-02-10T09:42:59.609047+0000 mon.b (mon.2) 361 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.602448+0000 mgr.x (mgr.34700) 108 : audit [DBG] from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.604515+0000 mon.b (mon.2) 351 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.604887+0000 mon.b (mon.2) 352 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.605203+0000 mon.b (mon.2) 353 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.605505+0000 mon.b (mon.2) 354 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.605792+0000 mon.b (mon.2) 355 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.606072+0000 mon.b (mon.2) 356 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.607875+0000 mon.b (mon.2) 357 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.608194+0000 mon.b (mon.2) 358 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.608490+0000 mon.b (mon.2) 359 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.608770+0000 mon.b (mon.2) 360 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:00 smithi153 bash[39117]: audit 2024-02-10T09:42:59.609047+0000 mon.b (mon.2) 361 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.602448+0000 mgr.x (mgr.34700) 108 : audit [DBG] from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.604515+0000 mon.b (mon.2) 351 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.604887+0000 mon.b (mon.2) 352 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.605203+0000 mon.b (mon.2) 353 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.605505+0000 mon.b (mon.2) 354 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.605792+0000 mon.b (mon.2) 355 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.606072+0000 mon.b (mon.2) 356 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.607875+0000 mon.b (mon.2) 357 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.608194+0000 mon.b (mon.2) 358 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.608490+0000 mon.b (mon.2) 359 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.608770+0000 mon.b (mon.2) 360 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:00 smithi204 bash[35809]: audit 2024-02-10T09:42:59.609047+0000 mon.b (mon.2) 361 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.920834+0000 mon.b (mon.2) 362 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-10T09:43:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.921053+0000 mon.b (mon.2) 363 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.921244+0000 mon.b (mon.2) 364 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.921395+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.921429+0000 mon.b (mon.2) 365 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.921804+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.921957+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: audit 2024-02-10T09:43:00.922109+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-10T09:43:01.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:01 smithi204 bash[35809]: cluster 2024-02-10T09:43:00.951541+0000 mgr.x (mgr.34700) 109 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.920834+0000 mon.b (mon.2) 362 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.921053+0000 mon.b (mon.2) 363 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.921244+0000 mon.b (mon.2) 364 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.921395+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.921429+0000 mon.b (mon.2) 365 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.921804+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.921957+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: audit 2024-02-10T09:43:00.922109+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-10T09:43:01.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[39117]: cluster 2024-02-10T09:43:00.951541+0000 mgr.x (mgr.34700) 109 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.920834+0000 mon.b (mon.2) 362 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.921053+0000 mon.b (mon.2) 363 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.921244+0000 mon.b (mon.2) 364 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.921395+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.921429+0000 mon.b (mon.2) 365 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.921804+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.921957+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: audit 2024-02-10T09:43:00.922109+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-10T09:43:01.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:01 smithi153 bash[35780]: cluster 2024-02-10T09:43:00.951541+0000 mgr.x (mgr.34700) 109 : cluster [DBG] pgmap v98: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:02.246 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:43:02.246 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:43:02.246 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:43:02.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:43:02.247 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:43:02.248 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:43:02.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-10T09:43:03.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:02 smithi153 bash[35780]: audit 2024-02-10T09:43:02.246104+0000 mon.a (mon.0) 855 : audit [DBG] from='client.? 172.21.15.153:0/3008052237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:03.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:02 smithi153 bash[39117]: audit 2024-02-10T09:43:02.246104+0000 mon.a (mon.0) 855 : audit [DBG] from='client.? 172.21.15.153:0/3008052237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:02 smithi204 bash[35809]: audit 2024-02-10T09:43:02.246104+0000 mon.a (mon.0) 855 : audit [DBG] from='client.? 172.21.15.153:0/3008052237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:03.894 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:03 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:03] "GET /metrics HTTP/1.1" 200 153265 "" "Prometheus/2.33.4" 2024-02-10T09:43:04.176 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:03 smithi153 bash[35780]: cluster 2024-02-10T09:43:02.952271+0000 mgr.x (mgr.34700) 110 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:04.176 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:03 smithi153 bash[39117]: cluster 2024-02-10T09:43:02.952271+0000 mgr.x (mgr.34700) 110 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:04.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:03 smithi204 bash[35809]: cluster 2024-02-10T09:43:02.952271+0000 mgr.x (mgr.34700) 110 : cluster [DBG] pgmap v99: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:04.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:43:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:05 smithi204 bash[35809]: audit 2024-02-10T09:43:04.875609+0000 mgr.x (mgr.34700) 111 : audit [DBG] from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:05 smithi204 bash[35809]: cluster 2024-02-10T09:43:04.952777+0000 mgr.x (mgr.34700) 112 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:05.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:05 smithi153 bash[35780]: audit 2024-02-10T09:43:04.875609+0000 mgr.x (mgr.34700) 111 : audit [DBG] from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:05.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:05 smithi153 bash[35780]: cluster 2024-02-10T09:43:04.952777+0000 mgr.x (mgr.34700) 112 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:05.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:05 smithi153 bash[39117]: audit 2024-02-10T09:43:04.875609+0000 mgr.x (mgr.34700) 111 : audit [DBG] from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:05.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:05 smithi153 bash[39117]: cluster 2024-02-10T09:43:04.952777+0000 mgr.x (mgr.34700) 112 : cluster [DBG] pgmap v100: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:06.870 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.865298+0000 mon.b (mon.2) 366 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.865856+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.868914+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:43:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.875633+0000 mon.b (mon.2) 367 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.878593+0000 mon.b (mon.2) 368 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.879293+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[39117]: audit 2024-02-10T09:43:06.884709+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.865298+0000 mon.b (mon.2) 366 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.865856+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.868914+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.875633+0000 mon.b (mon.2) 367 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.878593+0000 mon.b (mon.2) 368 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.879293+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:06 smithi153 bash[35780]: audit 2024-02-10T09:43:06.884709+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:07.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.865298+0000 mon.b (mon.2) 366 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.865856+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.868914+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:43:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.875633+0000 mon.b (mon.2) 367 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.878593+0000 mon.b (mon.2) 368 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.879293+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:07.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:06 smithi204 bash[35809]: audit 2024-02-10T09:43:06.884709+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:07.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:43:07.927 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[35780]: cephadm 2024-02-10T09:43:06.864389+0000 mgr.x (mgr.34700) 113 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:07.927 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[35780]: cluster 2024-02-10T09:43:06.877838+0000 mgr.x (mgr.34700) 114 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:07.927 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[35780]: cluster 2024-02-10T09:43:06.878078+0000 mgr.x (mgr.34700) 115 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:07.927 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[35780]: cephadm 2024-02-10T09:43:06.889308+0000 mgr.x (mgr.34700) 116 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[35780]: cephadm 2024-02-10T09:43:06.895772+0000 mgr.x (mgr.34700) 117 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[39117]: cephadm 2024-02-10T09:43:06.864389+0000 mgr.x (mgr.34700) 113 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[39117]: cluster 2024-02-10T09:43:06.877838+0000 mgr.x (mgr.34700) 114 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[39117]: cluster 2024-02-10T09:43:06.878078+0000 mgr.x (mgr.34700) 115 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[39117]: cephadm 2024-02-10T09:43:06.889308+0000 mgr.x (mgr.34700) 116 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:08.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:07 smithi153 bash[39117]: cephadm 2024-02-10T09:43:06.895772+0000 mgr.x (mgr.34700) 117 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:07 smithi204 bash[35809]: cephadm 2024-02-10T09:43:06.864389+0000 mgr.x (mgr.34700) 113 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:07 smithi204 bash[35809]: cluster 2024-02-10T09:43:06.877838+0000 mgr.x (mgr.34700) 114 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:08.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:07 smithi204 bash[35809]: cluster 2024-02-10T09:43:06.878078+0000 mgr.x (mgr.34700) 115 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:08.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:07 smithi204 bash[35809]: cephadm 2024-02-10T09:43:06.889308+0000 mgr.x (mgr.34700) 116 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:08.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:07 smithi204 bash[35809]: cephadm 2024-02-10T09:43:06.895772+0000 mgr.x (mgr.34700) 117 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:09.482 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:43:09.482 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-02-10T09:43:09.482 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:09.482 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-02-10T09:43:09.482 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:09.482 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:09.503 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: debug 2024-02-10T09:43:09.659+0000 7fc22683d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:43:09.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: main() 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:43:09.833 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:09.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: yield (conn, connr) 2024-02-10T09:43:09.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:09.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: code, '\n'.join(err))) 2024-02-10T09:43:09.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:09.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:09.834 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Reconfig daemon grafana.a ... 2024-02-10T09:43:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:09.834 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[54926]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:09 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:09.835 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: Traceback (most recent call last): 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: main() 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: r = ctx.func(ctx) 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: return func(ctx) 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: ports=daemon_ports) 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:09.836 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:09.837 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:09.837 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:09.837 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[44646]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:10.141 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:43:10.141 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (11m) 3m ago 12m 25.9M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:43:10.141 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 error 3m ago 14m 35.2M - 6.6.2 87a51ecf0b1c 72f83803a230 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 running (16m) 3m ago 24m 412M - 16.2.14-521-g0e714d9a 1fbc92c859dc 46c6073a79be 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (7m) 3m ago 26m 382M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (26m) 3m ago 26m 173M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (24m) 3m ago 24m 148M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (25m) 3m ago 25m 146M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 error 3m ago 16m - - 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 error 3m ago 16m - - 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (23m) 3m ago 24m 43.4M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (23m) 3m ago 23m 44.2M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (23m) 3m ago 23m 42.7M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (23m) 3m ago 23m 45.7M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (22m) 3m ago 22m 47.5M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (22m) 3m ago 22m 48.3M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (22m) 3m ago 22m 46.2M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:43:10.142 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (22m) 3m ago 22m 45.4M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:43:10.143 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (11m) 3m ago 11m 63.8M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:43:10.143 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (14m) 3m ago 14m 34.8M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:43:10.143 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (14m) 3m ago 14m 36.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:43:10.188 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:09 smithi153 bash[35780]: cluster 2024-02-10T09:43:08.878346+0000 mgr.x (mgr.34700) 118 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:10.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:09 smithi153 bash[39117]: cluster 2024-02-10T09:43:08.878346+0000 mgr.x (mgr.34700) 118 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:10.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:09 smithi204 bash[35809]: cluster 2024-02-10T09:43:08.878346+0000 mgr.x (mgr.34700) 118 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:43:10.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:43:10.525 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "", 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 image", 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:43:10.838 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:43:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:09.491424+0000 mgr.x (mgr.34700) 119 : audit [DBG] from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: cephadm 2024-02-10T09:43:09.660217+0000 mgr.x (mgr.34700) 120 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: main() 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:11.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:09.491424+0000 mgr.x (mgr.34700) 119 : audit [DBG] from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: cephadm 2024-02-10T09:43:09.660217+0000 mgr.x (mgr.34700) 120 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: main() 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:43:11.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:43:11.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:43:11.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:11.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: main() 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: cephadm 2024-02-10T09:43:09.660942+0000 mgr.x (mgr.34700) 121 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:09.804355+0000 mgr.x (mgr.34700) 122 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.131385+0000 mon.b (mon.2) 369 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.131756+0000 mon.b (mon.2) 370 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.132068+0000 mon.b (mon.2) 371 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.132351+0000 mon.b (mon.2) 372 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.132638+0000 mon.b (mon.2) 373 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.132902+0000 mon.b (mon.2) 374 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.134573+0000 mon.b (mon.2) 375 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.134857+0000 mon.b (mon.2) 376 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.135121+0000 mon.b (mon.2) 377 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.135399+0000 mon.b (mon.2) 378 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.135652+0000 mon.b (mon.2) 379 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[35780]: audit 2024-02-10T09:43:10.523658+0000 mon.a (mon.0) 860 : audit [DBG] from='client.? 172.21.15.153:0/3819239912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: main() 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: cephadm 2024-02-10T09:43:09.660942+0000 mgr.x (mgr.34700) 121 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:09.804355+0000 mgr.x (mgr.34700) 122 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.131385+0000 mon.b (mon.2) 369 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.131756+0000 mon.b (mon.2) 370 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.132068+0000 mon.b (mon.2) 371 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.132351+0000 mon.b (mon.2) 372 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.132638+0000 mon.b (mon.2) 373 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.132902+0000 mon.b (mon.2) 374 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.134573+0000 mon.b (mon.2) 375 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.134857+0000 mon.b (mon.2) 376 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.135121+0000 mon.b (mon.2) 377 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.135399+0000 mon.b (mon.2) 378 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.135652+0000 mon.b (mon.2) 379 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.200 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:10 smithi153 bash[39117]: audit 2024-02-10T09:43:10.523658+0000 mon.a (mon.0) 860 : audit [DBG] from='client.? 172.21.15.153:0/3819239912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:09.491424+0000 mgr.x (mgr.34700) 119 : audit [DBG] from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: cephadm 2024-02-10T09:43:09.660217+0000 mgr.x (mgr.34700) 120 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:11.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: main() 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:11.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: main() 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:43:11.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: cephadm 2024-02-10T09:43:09.660942+0000 mgr.x (mgr.34700) 121 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:09.804355+0000 mgr.x (mgr.34700) 122 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.131385+0000 mon.b (mon.2) 369 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.131756+0000 mon.b (mon.2) 370 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.132068+0000 mon.b (mon.2) 371 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.132351+0000 mon.b (mon.2) 372 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.132638+0000 mon.b (mon.2) 373 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.132902+0000 mon.b (mon.2) 374 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.134573+0000 mon.b (mon.2) 375 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.134857+0000 mon.b (mon.2) 376 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.135121+0000 mon.b (mon.2) 377 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.135399+0000 mon.b (mon.2) 378 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.135652+0000 mon.b (mon.2) 379 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:11.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:10 smithi204 bash[35809]: audit 2024-02-10T09:43:10.523658+0000 mon.a (mon.0) 860 : audit [DBG] from='client.? 172.21.15.153:0/3819239912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:10.129518+0000 mgr.x (mgr.34700) 123 : audit [DBG] from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:10.836956+0000 mgr.x (mgr.34700) 124 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: cluster 2024-02-10T09:43:10.878866+0000 mgr.x (mgr.34700) 125 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:12.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.529169+0000 mon.b (mon.2) 380 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.529797+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.534258+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.535870+0000 mon.b (mon.2) 381 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.536903+0000 mon.b (mon.2) 382 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.537315+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.542165+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.545964+0000 mon.b (mon.2) 383 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.546294+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: cluster 2024-02-10T09:43:11.552601+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.556109+0000 mon.b (mon.2) 384 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.556623+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.557389+0000 mon.b (mon.2) 385 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:43:12.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:11 smithi204 bash[35809]: audit 2024-02-10T09:43:11.557811+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:43:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:10.129518+0000 mgr.x (mgr.34700) 123 : audit [DBG] from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:10.836956+0000 mgr.x (mgr.34700) 124 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: cluster 2024-02-10T09:43:10.878866+0000 mgr.x (mgr.34700) 125 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:12.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.529169+0000 mon.b (mon.2) 380 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.529797+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.534258+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.535870+0000 mon.b (mon.2) 381 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.536903+0000 mon.b (mon.2) 382 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.537315+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:10.129518+0000 mgr.x (mgr.34700) 123 : audit [DBG] from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:10.836956+0000 mgr.x (mgr.34700) 124 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: cluster 2024-02-10T09:43:10.878866+0000 mgr.x (mgr.34700) 125 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 6.1 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.529169+0000 mon.b (mon.2) 380 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:12.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.529797+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.534258+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mgr\"], \"hosts\": null, \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.535870+0000 mon.b (mon.2) 381 : audit [DBG] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.536903+0000 mon.b (mon.2) 382 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.537315+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.542165+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.545964+0000 mon.b (mon.2) 383 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.546294+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: cluster 2024-02-10T09:43:11.552601+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.556109+0000 mon.b (mon.2) 384 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.556623+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:43:12.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.557389+0000 mon.b (mon.2) 385 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[35780]: audit 2024-02-10T09:43:11.557811+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.542165+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.545964+0000 mon.b (mon.2) 383 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.546294+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: cluster 2024-02-10T09:43:11.552601+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.556109+0000 mon.b (mon.2) 384 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.556623+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.557389+0000 mon.b (mon.2) 385 : audit [INF] from='mgr.34700 172.21.15.204:0/3492830394' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:43:12.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:11 smithi153 bash[39117]: audit 2024-02-10T09:43:11.557811+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:43:12.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[44646]: debug 2024-02-10T09:43:12.543+0000 7fc28c23e700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T09:43:12.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[44646]: ignoring --setuser ceph since I am not root 2024-02-10T09:43:12.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[44646]: ignoring --setgroup ceph since I am not root 2024-02-10T09:43:12.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[44646]: debug 2024-02-10T09:43:12.587+0000 7f5365136700 1 -- 172.21.15.204:0/823494451 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5571ea2e24e0 con 0x5571ea2b9000 2024-02-10T09:43:12.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[44646]: debug 2024-02-10T09:43:12.709+0000 7f5375314680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:43:12.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[44646]: debug 2024-02-10T09:43:12.790+0000 7f5375314680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:43:13.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:13 smithi204 bash[44646]: debug 2024-02-10T09:43:13.155+0000 7f5375314680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cephadm 2024-02-10T09:43:11.534723+0000 mgr.x (mgr.34700) 126 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cephadm 2024-02-10T09:43:11.534810+0000 mgr.x (mgr.34700) 127 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cephadm 2024-02-10T09:43:11.543036+0000 mgr.x (mgr.34700) 128 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cephadm 2024-02-10T09:43:11.543551+0000 mgr.x (mgr.34700) 129 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cephadm 2024-02-10T09:43:11.545691+0000 mgr.x (mgr.34700) 130 : cephadm [INF] Failing over to other MGR 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: audit 2024-02-10T09:43:12.543578+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cluster 2024-02-10T09:43:12.543643+0000 mon.a (mon.0) 870 : cluster [DBG] mgrmap e35: y(active, starting, since 0.996914s) 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: audit 2024-02-10T09:43:12.555565+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:43:13.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:12 smithi204 bash[35809]: cluster 2024-02-10T09:43:12.555617+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cephadm 2024-02-10T09:43:11.534723+0000 mgr.x (mgr.34700) 126 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cephadm 2024-02-10T09:43:11.534810+0000 mgr.x (mgr.34700) 127 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cephadm 2024-02-10T09:43:11.543036+0000 mgr.x (mgr.34700) 128 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cephadm 2024-02-10T09:43:11.543551+0000 mgr.x (mgr.34700) 129 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cephadm 2024-02-10T09:43:11.545691+0000 mgr.x (mgr.34700) 130 : cephadm [INF] Failing over to other MGR 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: audit 2024-02-10T09:43:12.543578+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cluster 2024-02-10T09:43:12.543643+0000 mon.a (mon.0) 870 : cluster [DBG] mgrmap e35: y(active, starting, since 0.996914s) 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: audit 2024-02-10T09:43:12.555565+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:43:13.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[35780]: cluster 2024-02-10T09:43:12.555617+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cephadm 2024-02-10T09:43:11.534723+0000 mgr.x (mgr.34700) 126 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cephadm 2024-02-10T09:43:11.534810+0000 mgr.x (mgr.34700) 127 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cephadm 2024-02-10T09:43:11.543036+0000 mgr.x (mgr.34700) 128 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cephadm 2024-02-10T09:43:11.543551+0000 mgr.x (mgr.34700) 129 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cephadm 2024-02-10T09:43:11.545691+0000 mgr.x (mgr.34700) 130 : cephadm [INF] Failing over to other MGR 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: audit 2024-02-10T09:43:12.543578+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cluster 2024-02-10T09:43:12.543643+0000 mon.a (mon.0) 870 : cluster [DBG] mgrmap e35: y(active, starting, since 0.996914s) 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: audit 2024-02-10T09:43:12.555565+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.34700 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:43:13.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:12 smithi153 bash[39117]: cluster 2024-02-10T09:43:12.555617+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:43:14.208 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:13 smithi204 bash[44646]: debug 2024-02-10T09:43:13.909+0000 7f5375314680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:43:14.208 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:13 smithi204 bash[44646]: debug 2024-02-10T09:43:13.994+0000 7f5375314680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:43:14.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:14 smithi204 bash[44646]: debug 2024-02-10T09:43:14.206+0000 7f5375314680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:43:14.689 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:43:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:43:14.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:43:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:43:15.459 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:15 smithi204 bash[44646]: debug 2024-02-10T09:43:15.155+0000 7f5375314680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:43:15.459 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:15 smithi204 bash[44646]: debug 2024-02-10T09:43:15.285+0000 7f5375314680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:43:15.459 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:15 smithi204 bash[44646]: debug 2024-02-10T09:43:15.366+0000 7f5375314680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:43:15.459 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:15 smithi204 bash[44646]: debug 2024-02-10T09:43:15.457+0000 7f5375314680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:43:15.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:15 smithi204 bash[44646]: debug 2024-02-10T09:43:15.654+0000 7f5375314680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:43:16.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:15 smithi204 bash[44646]: debug 2024-02-10T09:43:15.938+0000 7f5375314680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:43:16.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:16 smithi204 bash[44646]: debug 2024-02-10T09:43:16.036+0000 7f5375314680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:43:17.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:16 smithi204 bash[44646]: debug 2024-02-10T09:43:16.774+0000 7f5375314680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:43:17.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:16 smithi204 bash[44646]: debug 2024-02-10T09:43:16.858+0000 7f5375314680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:43:17.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:16 smithi204 bash[44646]: debug 2024-02-10T09:43:16.952+0000 7f5375314680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:43:17.394 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: debug 2024-02-10T09:43:17.130+0000 7f5375314680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:43:17.665 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: debug 2024-02-10T09:43:17.392+0000 7f5375314680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:43:17.665 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: debug 2024-02-10T09:43:17.526+0000 7f5375314680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:43:17.925 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: debug 2024-02-10T09:43:17.663+0000 7f5375314680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:43:17.925 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: debug 2024-02-10T09:43:17.837+0000 7f5375314680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:43:18.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[35809]: cluster 2024-02-10T09:43:17.938945+0000 mon.a (mon.0) 873 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[35809]: audit 2024-02-10T09:43:17.939291+0000 mon.b (mon.2) 386 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[35809]: audit 2024-02-10T09:43:17.939577+0000 mon.b (mon.2) 387 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[35809]: audit 2024-02-10T09:43:17.941889+0000 mon.b (mon.2) 388 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[35809]: audit 2024-02-10T09:43:17.942814+0000 mon.b (mon.2) 389 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: debug 2024-02-10T09:43:17.924+0000 7f5375314680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:17 smithi204 bash[44646]: [10/Feb/2024:09:43:17] ENGINE Bus STARTING 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[44646]: CherryPy Checker: 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[44646]: The Application mounted at '' has an empty config. 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[44646]: [10/Feb/2024:09:43:18] ENGINE Serving on http://:::9283 2024-02-10T09:43:18.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:18 smithi204 bash[44646]: [10/Feb/2024:09:43:18] ENGINE Bus STARTED 2024-02-10T09:43:18.439 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:43:18.089Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=5 2024-02-10T09:43:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:17 smithi153 bash[35780]: cluster 2024-02-10T09:43:17.938945+0000 mon.a (mon.0) 873 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:43:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:17 smithi153 bash[35780]: audit 2024-02-10T09:43:17.939291+0000 mon.b (mon.2) 386 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:43:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:17 smithi153 bash[35780]: audit 2024-02-10T09:43:17.939577+0000 mon.b (mon.2) 387 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[35780]: audit 2024-02-10T09:43:17.941889+0000 mon.b (mon.2) 388 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:43:18.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[35780]: audit 2024-02-10T09:43:17.942814+0000 mon.b (mon.2) 389 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:18.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[39117]: cluster 2024-02-10T09:43:17.938945+0000 mon.a (mon.0) 873 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:43:18.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[39117]: audit 2024-02-10T09:43:17.939291+0000 mon.b (mon.2) 386 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:43:18.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[39117]: audit 2024-02-10T09:43:17.939577+0000 mon.b (mon.2) 387 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:18.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[39117]: audit 2024-02-10T09:43:17.941889+0000 mon.b (mon.2) 388 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:43:18.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:18 smithi153 bash[39117]: audit 2024-02-10T09:43:17.942814+0000 mon.b (mon.2) 389 : audit [DBG] from='mgr.? 172.21.15.204:0/1373524464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:18.820 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:43:18] ENGINE Bus STOPPING 2024-02-10T09:43:18.820 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:43:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:43:18.820 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:43:18] ENGINE Bus STOPPED 2024-02-10T09:43:18.820 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:43:18] ENGINE Bus STARTING 2024-02-10T09:43:18.820 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [80B blob data] 2024-02-10T09:43:18.820 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [80B blob data] 2024-02-10T09:43:19.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: CherryPy Checker: 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: The Application mounted at '' has an empty config. 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:43:18] ENGINE Serving on http://:::9283 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:43:18] ENGINE Bus STARTED 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: cluster 2024-02-10T09:43:17.997056+0000 mon.a (mon.0) 874 : cluster [DBG] mgrmap e36: y(active, starting, since 6s), standbys: x 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:17.997122+0000 mon.b (mon.2) 390 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:17.997341+0000 mon.b (mon.2) 391 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.547620+0000 mon.c (mon.1) 140 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:43:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.547858+0000 mon.c (mon.1) 141 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.548053+0000 mon.c (mon.1) 142 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.549856+0000 mon.c (mon.1) 143 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.550171+0000 mon.c (mon.1) 144 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.550460+0000 mon.c (mon.1) 145 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.550754+0000 mon.c (mon.1) 146 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.551017+0000 mon.c (mon.1) 147 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.551293+0000 mon.c (mon.1) 148 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.551556+0000 mon.c (mon.1) 149 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.551846+0000 mon.c (mon.1) 150 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.552134+0000 mon.c (mon.1) 151 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.552337+0000 mon.c (mon.1) 152 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.552883+0000 mon.c (mon.1) 153 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:43:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.553698+0000 mon.c (mon.1) 154 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: cluster 2024-02-10T09:43:17.997056+0000 mon.a (mon.0) 874 : cluster [DBG] mgrmap e36: y(active, starting, since 6s), standbys: x 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:17.997122+0000 mon.b (mon.2) 390 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:17.997341+0000 mon.b (mon.2) 391 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.547620+0000 mon.c (mon.1) 140 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.547858+0000 mon.c (mon.1) 141 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.548053+0000 mon.c (mon.1) 142 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.549856+0000 mon.c (mon.1) 143 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.550171+0000 mon.c (mon.1) 144 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.550460+0000 mon.c (mon.1) 145 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.550754+0000 mon.c (mon.1) 146 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:43:19.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.551017+0000 mon.c (mon.1) 147 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.551293+0000 mon.c (mon.1) 148 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.551556+0000 mon.c (mon.1) 149 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.551846+0000 mon.c (mon.1) 150 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.552134+0000 mon.c (mon.1) 151 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.552337+0000 mon.c (mon.1) 152 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.552883+0000 mon.c (mon.1) 153 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.553698+0000 mon.c (mon.1) 154 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.554754+0000 mon.c (mon.1) 155 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.555125+0000 mon.c (mon.1) 156 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.555450+0000 mon.c (mon.1) 157 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.555783+0000 mon.c (mon.1) 158 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.556129+0000 mon.c (mon.1) 159 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:43:19.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.556463+0000 mon.c (mon.1) 160 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.556773+0000 mon.c (mon.1) 161 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.557103+0000 mon.c (mon.1) 162 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.557435+0000 mon.c (mon.1) 163 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.557786+0000 mon.c (mon.1) 164 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.558120+0000 mon.c (mon.1) 165 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.558434+0000 mon.c (mon.1) 166 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.558770+0000 mon.c (mon.1) 167 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.559063+0000 mon.c (mon.1) 168 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.559352+0000 mon.c (mon.1) 169 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.559654+0000 mon.c (mon.1) 170 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:43:19.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.560019+0000 mon.c (mon.1) 171 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.554754+0000 mon.c (mon.1) 155 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.555125+0000 mon.c (mon.1) 156 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.555450+0000 mon.c (mon.1) 157 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.555783+0000 mon.c (mon.1) 158 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.556129+0000 mon.c (mon.1) 159 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.556463+0000 mon.c (mon.1) 160 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.556773+0000 mon.c (mon.1) 161 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.557103+0000 mon.c (mon.1) 162 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.557435+0000 mon.c (mon.1) 163 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.557786+0000 mon.c (mon.1) 164 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.558120+0000 mon.c (mon.1) 165 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.558434+0000 mon.c (mon.1) 166 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.558770+0000 mon.c (mon.1) 167 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:43:19.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.559063+0000 mon.c (mon.1) 168 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.559352+0000 mon.c (mon.1) 169 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.559654+0000 mon.c (mon.1) 170 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.560019+0000 mon.c (mon.1) 171 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.560345+0000 mon.c (mon.1) 172 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.560648+0000 mon.c (mon.1) 173 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.560971+0000 mon.c (mon.1) 174 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.561268+0000 mon.c (mon.1) 175 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.561575+0000 mon.c (mon.1) 176 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.561913+0000 mon.c (mon.1) 177 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:43:19.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.562265+0000 mon.c (mon.1) 178 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.562605+0000 mon.c (mon.1) 179 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.562950+0000 mon.c (mon.1) 180 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: cluster 2024-02-10T09:43:18.590265+0000 mon.a (mon.0) 875 : cluster [INF] Manager daemon y is now available 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.694624+0000 mon.c (mon.1) 181 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.696222+0000 mon.c (mon.1) 182 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.742362+0000 mon.c (mon.1) 183 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.742700+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.755706+0000 mon.c (mon.1) 184 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.756031+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.922739+0000 mon.c (mon.1) 185 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[35780]: audit 2024-02-10T09:43:18.923614+0000 mon.c (mon.1) 186 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.560345+0000 mon.c (mon.1) 172 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:43:19.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.560648+0000 mon.c (mon.1) 173 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.560971+0000 mon.c (mon.1) 174 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.561268+0000 mon.c (mon.1) 175 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.561575+0000 mon.c (mon.1) 176 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.561913+0000 mon.c (mon.1) 177 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.562265+0000 mon.c (mon.1) 178 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.562605+0000 mon.c (mon.1) 179 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.562950+0000 mon.c (mon.1) 180 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: cluster 2024-02-10T09:43:18.590265+0000 mon.a (mon.0) 875 : cluster [INF] Manager daemon y is now available 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.694624+0000 mon.c (mon.1) 181 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.696222+0000 mon.c (mon.1) 182 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.742362+0000 mon.c (mon.1) 183 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.742700+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.755706+0000 mon.c (mon.1) 184 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.756031+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:43:19.198 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.922739+0000 mon.c (mon.1) 185 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:19.199 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:19 smithi153 bash[39117]: audit 2024-02-10T09:43:18.923614+0000 mon.c (mon.1) 186 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: cluster 2024-02-10T09:43:17.997056+0000 mon.a (mon.0) 874 : cluster [DBG] mgrmap e36: y(active, starting, since 6s), standbys: x 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:17.997122+0000 mon.b (mon.2) 390 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:17.997341+0000 mon.b (mon.2) 391 : audit [DBG] from='mgr.? 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.547620+0000 mon.c (mon.1) 140 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.547858+0000 mon.c (mon.1) 141 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.548053+0000 mon.c (mon.1) 142 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.549856+0000 mon.c (mon.1) 143 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.550171+0000 mon.c (mon.1) 144 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:43:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.550460+0000 mon.c (mon.1) 145 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.550754+0000 mon.c (mon.1) 146 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.551017+0000 mon.c (mon.1) 147 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.551293+0000 mon.c (mon.1) 148 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.551556+0000 mon.c (mon.1) 149 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.551846+0000 mon.c (mon.1) 150 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.552134+0000 mon.c (mon.1) 151 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.552337+0000 mon.c (mon.1) 152 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.552883+0000 mon.c (mon.1) 153 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.553698+0000 mon.c (mon.1) 154 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.554754+0000 mon.c (mon.1) 155 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.555125+0000 mon.c (mon.1) 156 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.555450+0000 mon.c (mon.1) 157 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:43:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.555783+0000 mon.c (mon.1) 158 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.556129+0000 mon.c (mon.1) 159 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.556463+0000 mon.c (mon.1) 160 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.556773+0000 mon.c (mon.1) 161 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.557103+0000 mon.c (mon.1) 162 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.557435+0000 mon.c (mon.1) 163 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.557786+0000 mon.c (mon.1) 164 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.558120+0000 mon.c (mon.1) 165 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.558434+0000 mon.c (mon.1) 166 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.558770+0000 mon.c (mon.1) 167 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.559063+0000 mon.c (mon.1) 168 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.559352+0000 mon.c (mon.1) 169 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.559654+0000 mon.c (mon.1) 170 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.560019+0000 mon.c (mon.1) 171 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:43:19.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.560345+0000 mon.c (mon.1) 172 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.560648+0000 mon.c (mon.1) 173 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.560971+0000 mon.c (mon.1) 174 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.561268+0000 mon.c (mon.1) 175 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.561575+0000 mon.c (mon.1) 176 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.561913+0000 mon.c (mon.1) 177 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.562265+0000 mon.c (mon.1) 178 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.562605+0000 mon.c (mon.1) 179 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.562950+0000 mon.c (mon.1) 180 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: cluster 2024-02-10T09:43:18.590265+0000 mon.a (mon.0) 875 : cluster [INF] Manager daemon y is now available 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.694624+0000 mon.c (mon.1) 181 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.696222+0000 mon.c (mon.1) 182 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.742362+0000 mon.c (mon.1) 183 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.742700+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.755706+0000 mon.c (mon.1) 184 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.756031+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:43:19.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.922739+0000 mon.c (mon.1) 185 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:19.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:19 smithi204 bash[35809]: audit 2024-02-10T09:43:18.923614+0000 mon.c (mon.1) 186 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:19.818 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:43:19.818 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:43:19.818 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:19.819 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:20.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:20 smithi204 bash[35809]: cluster 2024-02-10T09:43:19.015874+0000 mon.a (mon.0) 878 : cluster [DBG] mgrmap e37: y(active, since 7s), standbys: x 2024-02-10T09:43:20.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:20 smithi204 bash[35809]: cluster 2024-02-10T09:43:19.022147+0000 mgr.y (mgr.34736) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:20.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:20.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:20.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:20.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:20.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:20.083 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:19 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:20 smithi153 bash[39117]: cluster 2024-02-10T09:43:19.015874+0000 mon.a (mon.0) 878 : cluster [DBG] mgrmap e37: y(active, since 7s), standbys: x 2024-02-10T09:43:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:20 smithi153 bash[39117]: cluster 2024-02-10T09:43:19.022147+0000 mgr.y (mgr.34736) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:20 smithi153 bash[35780]: cluster 2024-02-10T09:43:19.015874+0000 mon.a (mon.0) 878 : cluster [DBG] mgrmap e37: y(active, since 7s), standbys: x 2024-02-10T09:43:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:20 smithi153 bash[35780]: cluster 2024-02-10T09:43:19.022147+0000 mgr.y (mgr.34736) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:20.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:20 smithi204 bash[55201]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:20.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:20 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:20.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:20 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:20.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:20 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:21 smithi204 bash[35809]: cluster 2024-02-10T09:43:20.010762+0000 mon.a (mon.0) 879 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:43:21.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:21 smithi204 bash[35809]: cluster 2024-02-10T09:43:20.010790+0000 mon.a (mon.0) 880 : cluster [INF] Cluster is now healthy 2024-02-10T09:43:21.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:21 smithi204 bash[35809]: cluster 2024-02-10T09:43:20.021481+0000 mon.a (mon.0) 881 : cluster [DBG] mgrmap e38: y(active, since 8s), standbys: x 2024-02-10T09:43:21.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:21 smithi204 bash[35809]: cluster 2024-02-10T09:43:20.551703+0000 mgr.y (mgr.34736) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[35780]: cluster 2024-02-10T09:43:20.010762+0000 mon.a (mon.0) 879 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:43:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[35780]: cluster 2024-02-10T09:43:20.010790+0000 mon.a (mon.0) 880 : cluster [INF] Cluster is now healthy 2024-02-10T09:43:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[35780]: cluster 2024-02-10T09:43:20.021481+0000 mon.a (mon.0) 881 : cluster [DBG] mgrmap e38: y(active, since 8s), standbys: x 2024-02-10T09:43:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[35780]: cluster 2024-02-10T09:43:20.551703+0000 mgr.y (mgr.34736) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:21.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[39117]: cluster 2024-02-10T09:43:20.010762+0000 mon.a (mon.0) 879 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-02-10T09:43:21.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[39117]: cluster 2024-02-10T09:43:20.010790+0000 mon.a (mon.0) 880 : cluster [INF] Cluster is now healthy 2024-02-10T09:43:21.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[39117]: cluster 2024-02-10T09:43:20.021481+0000 mon.a (mon.0) 881 : cluster [DBG] mgrmap e38: y(active, since 8s), standbys: x 2024-02-10T09:43:21.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:21 smithi153 bash[39117]: cluster 2024-02-10T09:43:20.551703+0000 mgr.y (mgr.34736) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:23.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.490821+0000 mon.c (mon.1) 187 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:23.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.490821+0000 mon.c (mon.1) 187 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:23.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.491321+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.491321+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.494427+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.774293+0000 mon.c (mon.1) 188 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.774664+0000 mon.c (mon.1) 189 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.774987+0000 mon.c (mon.1) 190 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.775261+0000 mon.c (mon.1) 191 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.775524+0000 mon.c (mon.1) 192 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.775782+0000 mon.c (mon.1) 193 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.776040+0000 mon.c (mon.1) 194 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.776278+0000 mon.c (mon.1) 195 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[35780]: audit 2024-02-10T09:43:22.776520+0000 mon.c (mon.1) 196 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.494427+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.774293+0000 mon.c (mon.1) 188 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.774664+0000 mon.c (mon.1) 189 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.774987+0000 mon.c (mon.1) 190 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.775261+0000 mon.c (mon.1) 191 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.775524+0000 mon.c (mon.1) 192 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.775782+0000 mon.c (mon.1) 193 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.776040+0000 mon.c (mon.1) 194 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.776278+0000 mon.c (mon.1) 195 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:22 smithi153 bash[39117]: audit 2024-02-10T09:43:22.776520+0000 mon.c (mon.1) 196 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.490821+0000 mon.c (mon.1) 187 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.491321+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.494427+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489316Z\", \"created\": \"2024-02-10T09:26:14.154248Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:13.634461Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:22.489546Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.774293+0000 mon.c (mon.1) 188 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.774664+0000 mon.c (mon.1) 189 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.774987+0000 mon.c (mon.1) 190 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.775261+0000 mon.c (mon.1) 191 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.775524+0000 mon.c (mon.1) 192 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.775782+0000 mon.c (mon.1) 193 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.776040+0000 mon.c (mon.1) 194 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.776278+0000 mon.c (mon.1) 195 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:23.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:22 smithi204 bash[35809]: audit 2024-02-10T09:43:22.776520+0000 mon.c (mon.1) 196 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:23.901 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:43:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: cluster 2024-02-10T09:43:22.552069+0000 mgr.y (mgr.34736) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:24.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: cluster 2024-02-10T09:43:22.552069+0000 mgr.y (mgr.34736) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.390668+0000 mon.c (mon.1) 197 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:24.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.390668+0000 mon.c (mon.1) 197 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:24.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.391269+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:24.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.395008+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:24.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.391269+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:24.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.395008+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:24.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.675339+0000 mon.c (mon.1) 198 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.675799+0000 mon.c (mon.1) 199 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.676149+0000 mon.c (mon.1) 200 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.676483+0000 mon.c (mon.1) 201 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.676817+0000 mon.c (mon.1) 202 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.677138+0000 mon.c (mon.1) 203 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.677458+0000 mon.c (mon.1) 204 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.677839+0000 mon.c (mon.1) 205 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.678226+0000 mon.c (mon.1) 206 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.679836+0000 mon.c (mon.1) 207 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.680265+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[35780]: audit 2024-02-10T09:43:23.685836+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.675339+0000 mon.c (mon.1) 198 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.675799+0000 mon.c (mon.1) 199 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.676149+0000 mon.c (mon.1) 200 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.676483+0000 mon.c (mon.1) 201 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.676817+0000 mon.c (mon.1) 202 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.677138+0000 mon.c (mon.1) 203 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.677458+0000 mon.c (mon.1) 204 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.677839+0000 mon.c (mon.1) 205 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.678226+0000 mon.c (mon.1) 206 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.679836+0000 mon.c (mon.1) 207 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.680265+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:24.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:23 smithi153 bash[39117]: audit 2024-02-10T09:43:23.685836+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:24.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: cluster 2024-02-10T09:43:22.552069+0000 mgr.y (mgr.34736) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:24.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.390668+0000 mon.c (mon.1) 197 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:24.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.391269+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:24.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.395008+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"46c6073a79be\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 401080320, \"cpu_percentage\": \"4.24%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388740Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:26:56.123455Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:17.117287Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:23.389419Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.675339+0000 mon.c (mon.1) 198 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.675799+0000 mon.c (mon.1) 199 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.676149+0000 mon.c (mon.1) 200 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.676483+0000 mon.c (mon.1) 201 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.676817+0000 mon.c (mon.1) 202 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.677138+0000 mon.c (mon.1) 203 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.677458+0000 mon.c (mon.1) 204 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.677839+0000 mon.c (mon.1) 205 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.678226+0000 mon.c (mon.1) 206 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.679836+0000 mon.c (mon.1) 207 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.680265+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:24.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:23 smithi204 bash[35809]: audit 2024-02-10T09:43:23.685836+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:24] "GET /metrics HTTP/1.1" 200 32887 "" "Prometheus/2.33.4" 2024-02-10T09:43:25.134 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:24 smithi204 bash[35809]: cephadm 2024-02-10T09:43:23.691586+0000 mgr.y (mgr.34736) 5 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:25.134 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:24 smithi204 bash[35809]: cephadm 2024-02-10T09:43:23.729982+0000 mgr.y (mgr.34736) 6 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:25.134 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:24 smithi204 bash[35809]: cluster 2024-02-10T09:43:24.494317+0000 mon.a (mon.0) 888 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:43:25.156 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:24 smithi153 bash[39117]: cephadm 2024-02-10T09:43:23.691586+0000 mgr.y (mgr.34736) 5 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:25.156 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:24 smithi153 bash[39117]: cephadm 2024-02-10T09:43:23.729982+0000 mgr.y (mgr.34736) 6 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:25.156 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:24 smithi153 bash[39117]: cluster 2024-02-10T09:43:24.494317+0000 mon.a (mon.0) 888 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:43:25.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:24 smithi153 bash[35780]: cephadm 2024-02-10T09:43:23.691586+0000 mgr.y (mgr.34736) 5 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:25.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:24 smithi153 bash[35780]: cephadm 2024-02-10T09:43:23.729982+0000 mgr.y (mgr.34736) 6 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:25.157 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:24 smithi153 bash[35780]: cluster 2024-02-10T09:43:24.494317+0000 mon.a (mon.0) 888 : cluster [WRN] Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:43:25.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:25 smithi153 bash[66122]: time="2024-02-10T09:43:25Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:43:26.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:26 smithi204 bash[54417]: time="2024-02-10T09:43:26Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:26.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:26.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:26.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 bash[55910]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:26.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:26.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:26.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:26 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:26.911 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[35780]: cluster 2024-02-10T09:43:25.680108+0000 mgr.y (mgr.34736) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:26.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[35780]: audit 2024-02-10T09:43:26.532015+0000 mon.c (mon.1) 208 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:26.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[35780]: audit 2024-02-10T09:43:26.533289+0000 mon.c (mon.1) 209 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:26.912 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[35780]: audit 2024-02-10T09:43:26.533635+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:26.912 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:43:26.529+0000 7f12c2f4e700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:43:26.913 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:26.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:26.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:43:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[39117]: cluster 2024-02-10T09:43:25.680108+0000 mgr.y (mgr.34736) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[39117]: audit 2024-02-10T09:43:26.532015+0000 mon.c (mon.1) 208 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[39117]: audit 2024-02-10T09:43:26.533289+0000 mon.c (mon.1) 209 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:27.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:26 smithi153 bash[39117]: audit 2024-02-10T09:43:26.533635+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:27.312 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:26 smithi204 bash[35809]: cluster 2024-02-10T09:43:25.680108+0000 mgr.y (mgr.34736) 7 : cluster [DBG] pgmap v7: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:27.312 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:26 smithi204 bash[35809]: audit 2024-02-10T09:43:26.532015+0000 mon.c (mon.1) 208 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:27.312 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:26 smithi204 bash[35809]: audit 2024-02-10T09:43:26.533289+0000 mon.c (mon.1) 209 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:27.313 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:26 smithi204 bash[35809]: audit 2024-02-10T09:43:26.533635+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: cephadm 2024-02-10T09:43:26.530213+0000 mgr.y (mgr.34736) 8 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:43:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: main() 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:28.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: cephadm 2024-02-10T09:43:26.530213+0000 mgr.y (mgr.34736) 8 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: main() 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:28.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:43:28.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: main() 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:28.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: main() 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:28.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:43:28.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:28.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:28.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:28.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:28.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:27 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: cephadm 2024-02-10T09:43:26.530213+0000 mgr.y (mgr.34736) 8 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: main() 2024-02-10T09:43:28.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:28.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: main() 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:28.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:43:28.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:28.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:28.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:28.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:28.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:28.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:27 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: cluster 2024-02-10T09:43:27.680439+0000 mgr.y (mgr.34736) 9 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: cephadm 2024-02-10T09:43:27.940913+0000 mgr.y (mgr.34736) 10 : cephadm [INF] Upgrade: Updating mgr.x 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: audit 2024-02-10T09:43:27.941255+0000 mon.c (mon.1) 210 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: audit 2024-02-10T09:43:27.941535+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: audit 2024-02-10T09:43:27.942321+0000 mon.c (mon.1) 211 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: audit 2024-02-10T09:43:27.943063+0000 mon.c (mon.1) 212 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:43:29.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[39117]: audit 2024-02-10T09:43:27.943996+0000 mon.c (mon.1) 213 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.x","key": "container_image"}]: dispatch 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: cluster 2024-02-10T09:43:27.680439+0000 mgr.y (mgr.34736) 9 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: cephadm 2024-02-10T09:43:27.940913+0000 mgr.y (mgr.34736) 10 : cephadm [INF] Upgrade: Updating mgr.x 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: audit 2024-02-10T09:43:27.941255+0000 mon.c (mon.1) 210 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: audit 2024-02-10T09:43:27.941535+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: audit 2024-02-10T09:43:27.942321+0000 mon.c (mon.1) 211 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: audit 2024-02-10T09:43:27.943063+0000 mon.c (mon.1) 212 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:43:29.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:28 smithi153 bash[35780]: audit 2024-02-10T09:43:27.943996+0000 mon.c (mon.1) 213 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.x","key": "container_image"}]: dispatch 2024-02-10T09:43:29.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: cluster 2024-02-10T09:43:27.680439+0000 mgr.y (mgr.34736) 9 : cluster [DBG] pgmap v8: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:29.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: cephadm 2024-02-10T09:43:27.940913+0000 mgr.y (mgr.34736) 10 : cephadm [INF] Upgrade: Updating mgr.x 2024-02-10T09:43:29.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: audit 2024-02-10T09:43:27.941255+0000 mon.c (mon.1) 210 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:43:29.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: audit 2024-02-10T09:43:27.941535+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:43:29.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: audit 2024-02-10T09:43:27.942321+0000 mon.c (mon.1) 211 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:43:29.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: audit 2024-02-10T09:43:27.943063+0000 mon.c (mon.1) 212 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:43:29.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:28 smithi204 bash[35809]: audit 2024-02-10T09:43:27.943996+0000 mon.c (mon.1) 213 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.x","key": "container_image"}]: dispatch 2024-02-10T09:43:29.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:29 smithi204 systemd[1]: Stopping Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:29 smithi153 bash[35780]: cephadm 2024-02-10T09:43:27.943621+0000 mgr.y (mgr.34736) 11 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:43:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:29 smithi153 bash[39117]: cephadm 2024-02-10T09:43:27.943621+0000 mgr.y (mgr.34736) 11 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:43:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:29 smithi204 bash[35809]: cephadm 2024-02-10T09:43:27.943621+0000 mgr.y (mgr.34736) 11 : cephadm [INF] Deploying daemon mgr.x on smithi204 2024-02-10T09:43:31.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:30 smithi153 bash[35780]: cluster 2024-02-10T09:43:29.681320+0000 mgr.y (mgr.34736) 12 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:31.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:30 smithi153 bash[39117]: cluster 2024-02-10T09:43:29.681320+0000 mgr.y (mgr.34736) 12 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:31.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:30 smithi204 bash[35809]: cluster 2024-02-10T09:43:29.681320+0000 mgr.y (mgr.34736) 12 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:33.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:32 smithi153 bash[35780]: cluster 2024-02-10T09:43:31.681851+0000 mgr.y (mgr.34736) 13 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:33.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:32 smithi153 bash[39117]: cluster 2024-02-10T09:43:31.681851+0000 mgr.y (mgr.34736) 13 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:33.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:32 smithi204 bash[35809]: cluster 2024-02-10T09:43:31.681851+0000 mgr.y (mgr.34736) 13 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:33 smithi204 bash[44646]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:43:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:34] "GET /metrics HTTP/1.1" 200 153254 "" "Prometheus/2.33.4" 2024-02-10T09:43:35.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:34 smithi204 bash[35809]: cluster 2024-02-10T09:43:33.682313+0000 mgr.y (mgr.34736) 14 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:34 smithi153 bash[35780]: cluster 2024-02-10T09:43:33.682313+0000 mgr.y (mgr.34736) 14 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:35.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:34 smithi153 bash[39117]: cluster 2024-02-10T09:43:33.682313+0000 mgr.y (mgr.34736) 14 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:36.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:43:36.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:43:36.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:36.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:36.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:36.951 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 bash[56157]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:36.952 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:36 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:37.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:36 smithi204 bash[35809]: cluster 2024-02-10T09:43:35.683033+0000 mgr.y (mgr.34736) 15 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:37.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:36 smithi153 bash[35780]: cluster 2024-02-10T09:43:35.683033+0000 mgr.y (mgr.34736) 15 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:37.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:36 smithi153 bash[39117]: cluster 2024-02-10T09:43:35.683033+0000 mgr.y (mgr.34736) 15 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:37 smithi204 bash[35809]: cluster 2024-02-10T09:43:37.683430+0000 mgr.y (mgr.34736) 16 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:38.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:37 smithi153 bash[35780]: cluster 2024-02-10T09:43:37.683430+0000 mgr.y (mgr.34736) 16 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:38.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:37 smithi153 bash[39117]: cluster 2024-02-10T09:43:37.683430+0000 mgr.y (mgr.34736) 16 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:39.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:39 smithi204 bash[56135]: time="2024-02-10T09:43:39Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.x in 10 seconds, resorting to SIGKILL" 2024-02-10T09:43:40.205 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:40 smithi204 bash[56135]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.x 2024-02-10T09:43:40.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=137/n/a 2024-02-10T09:43:41.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:40 smithi204 bash[35809]: cluster 2024-02-10T09:43:39.684221+0000 mgr.y (mgr.34736) 17 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:41.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:40 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T09:43:41.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:40 smithi204 systemd[1]: Stopped Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:41.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:40 smithi204 systemd[1]: Starting Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:41.175 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:43:41.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:40 smithi153 bash[39117]: cluster 2024-02-10T09:43:39.684221+0000 mgr.y (mgr.34736) 17 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:41.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:40 smithi153 bash[35780]: cluster 2024-02-10T09:43:39.684221+0000 mgr.y (mgr.34736) 17 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:41.745 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:41 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:41.355+0000 7fcc54318700 1 -- 172.21.15.204:0/2068791866 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e796a04e0 con 0x559e79675000 2024-02-10T09:43:41.745 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:41 smithi204 bash[56348]: c26e3d68d44903b7218ac404c415a15ed3ab18ca640af64ab3b73b14c8eef8af 2024-02-10T09:43:41.746 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:41 smithi204 systemd[1]: Started Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:41.746 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:41 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:41.483+0000 7fcc64914680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:43:41.746 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:41 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:41.569+0000 7fcc64914680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (11m) 19s ago 12m 26.0M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 error 18s ago 14m 35.7M - 6.6.2 87a51ecf0b1c 72f83803a230 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 starting - - - - 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (7m) 19s ago 26m 412M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (26m) 19s ago 27m 198M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (25m) 18s ago 25m 189M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (25m) 19s ago 25m 185M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 error 19s ago 17m - - 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 error 18s ago 17m - - 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (24m) 19s ago 24m 43.8M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (24m) 19s ago 24m 44.4M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (24m) 19s ago 24m 43.0M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:43:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (23m) 19s ago 23m 46.0M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (23m) 18s ago 23m 47.7M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (23m) 18s ago 23m 48.7M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (22m) 18s ago 22m 46.5M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (22m) 18s ago 22m 45.5M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (11m) 18s ago 11m 65.9M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (14m) 19s ago 14m 35.7M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:43:41.784 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (14m) 18s ago 14m 36.9M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:43:42.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:41 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:41.986+0000 7fcc64914680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:43:42.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:41 smithi204 bash[35809]: audit 2024-02-10T09:43:41.503492+0000 mon.c (mon.1) 214 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:42.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:41 smithi204 bash[35809]: audit 2024-02-10T09:43:41.503990+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:42.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:41 smithi204 bash[35809]: audit 2024-02-10T09:43:41.506952+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:42.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:41 smithi204 bash[35809]: audit 2024-02-10T09:43:41.508337+0000 mon.c (mon.1) 215 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:42.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:41 smithi204 bash[35809]: audit 2024-02-10T09:43:41.509670+0000 mon.c (mon.1) 216 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:42.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[39117]: audit 2024-02-10T09:43:41.503492+0000 mon.c (mon.1) 214 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:42.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[35780]: audit 2024-02-10T09:43:41.503492+0000 mon.c (mon.1) 214 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:42.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[39117]: audit 2024-02-10T09:43:41.503990+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:42.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[35780]: audit 2024-02-10T09:43:41.503990+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:42.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[35780]: audit 2024-02-10T09:43:41.506952+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:42.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[35780]: audit 2024-02-10T09:43:41.508337+0000 mon.c (mon.1) 215 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:42.092 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[35780]: audit 2024-02-10T09:43:41.509670+0000 mon.c (mon.1) 216 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:42.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[39117]: audit 2024-02-10T09:43:41.506952+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.87%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388570Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": [8443, 9283]}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50111447, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388825Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388897Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.388967Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389035Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389106Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.38%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:23.389150Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38723911, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:23.389219Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69195530, \"cpu_percentage\": \"0.27%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:23.389288Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:42.092 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[39117]: audit 2024-02-10T09:43:41.508337+0000 mon.c (mon.1) 215 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:42.093 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:41 smithi153 bash[39117]: audit 2024-02-10T09:43:41.509670+0000 mon.c (mon.1) 216 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:43:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:43:42.149 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:43:42.149 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:43:42.149 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "", 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mgr daemons", 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:43:42.452 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:43:42.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.162928+0000 mgr.y (mgr.34736) 18 : audit [DBG] from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:42.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.471626+0000 mgr.y (mgr.34736) 19 : audit [DBG] from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:42.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: cluster 2024-02-10T09:43:41.684644+0000 mgr.y (mgr.34736) 20 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:42.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.771815+0000 mgr.y (mgr.34736) 21 : audit [DBG] from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:42.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.773555+0000 mon.c (mon.1) 217 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:42.762 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.773910+0000 mon.c (mon.1) 218 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.774193+0000 mon.c (mon.1) 219 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.774473+0000 mon.c (mon.1) 220 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.774738+0000 mon.c (mon.1) 221 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.775004+0000 mon.c (mon.1) 222 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.776510+0000 mon.c (mon.1) 223 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.776834+0000 mon.c (mon.1) 224 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.777121+0000 mon.c (mon.1) 225 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.777390+0000 mon.c (mon.1) 226 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:41.777653+0000 mon.c (mon.1) 227 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:42.763 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:42 smithi204 bash[35809]: audit 2024-02-10T09:43:42.147533+0000 mon.a (mon.0) 893 : audit [DBG] from='client.? 172.21.15.153:0/2184622521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:43.079 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:42 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:42.764+0000 7fcc64914680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:43:43.079 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:42 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:42.857+0000 7fcc64914680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:43:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.162928+0000 mgr.y (mgr.34736) 18 : audit [DBG] from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.471626+0000 mgr.y (mgr.34736) 19 : audit [DBG] from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: cluster 2024-02-10T09:43:41.684644+0000 mgr.y (mgr.34736) 20 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.771815+0000 mgr.y (mgr.34736) 21 : audit [DBG] from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.773555+0000 mon.c (mon.1) 217 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.773910+0000 mon.c (mon.1) 218 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.774193+0000 mon.c (mon.1) 219 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.774473+0000 mon.c (mon.1) 220 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.774738+0000 mon.c (mon.1) 221 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.775004+0000 mon.c (mon.1) 222 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.776510+0000 mon.c (mon.1) 223 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.776834+0000 mon.c (mon.1) 224 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.777121+0000 mon.c (mon.1) 225 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.777390+0000 mon.c (mon.1) 226 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:41.777653+0000 mon.c (mon.1) 227 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[35780]: audit 2024-02-10T09:43:42.147533+0000 mon.a (mon.0) 893 : audit [DBG] from='client.? 172.21.15.153:0/2184622521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.162928+0000 mgr.y (mgr.34736) 18 : audit [DBG] from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:43.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.471626+0000 mgr.y (mgr.34736) 19 : audit [DBG] from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: cluster 2024-02-10T09:43:41.684644+0000 mgr.y (mgr.34736) 20 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.771815+0000 mgr.y (mgr.34736) 21 : audit [DBG] from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.773555+0000 mon.c (mon.1) 217 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.773910+0000 mon.c (mon.1) 218 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.774193+0000 mon.c (mon.1) 219 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.774473+0000 mon.c (mon.1) 220 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.774738+0000 mon.c (mon.1) 221 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.775004+0000 mon.c (mon.1) 222 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.776510+0000 mon.c (mon.1) 223 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.776834+0000 mon.c (mon.1) 224 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.777121+0000 mon.c (mon.1) 225 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.777390+0000 mon.c (mon.1) 226 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:41.777653+0000 mon.c (mon.1) 227 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:43:43.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:42 smithi153 bash[39117]: audit 2024-02-10T09:43:42.147533+0000 mon.a (mon.0) 893 : audit [DBG] from='client.? 172.21.15.153:0/2184622521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:43.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:43.077+0000 7fcc64914680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:43:44.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:43 smithi204 bash[35809]: audit 2024-02-10T09:43:42.451509+0000 mgr.y (mgr.34736) 22 : audit [DBG] from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:43 smithi153 bash[35780]: audit 2024-02-10T09:43:42.451509+0000 mgr.y (mgr.34736) 22 : audit [DBG] from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:43 smithi153 bash[39117]: audit 2024-02-10T09:43:42.451509+0000 mgr.y (mgr.34736) 22 : audit [DBG] from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:43:44.406 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:44.152+0000 7fcc64914680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:43:44.407 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:44.309+0000 7fcc64914680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:43:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:44] "GET /metrics HTTP/1.1" 200 153254 "" "Prometheus/2.33.4" 2024-02-10T09:43:44.730 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:44.405+0000 7fcc64914680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:43:44.730 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:44.527+0000 7fcc64914680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:43:45.028 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:44 smithi204 bash[35809]: cluster 2024-02-10T09:43:43.685147+0000 mgr.y (mgr.34736) 23 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:45.029 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:44.728+0000 7fcc64914680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:43:45.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:44 smithi153 bash[35780]: cluster 2024-02-10T09:43:43.685147+0000 mgr.y (mgr.34736) 23 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:45.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:44 smithi153 bash[39117]: cluster 2024-02-10T09:43:43.685147+0000 mgr.y (mgr.34736) 23 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:45.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:45.027+0000 7fcc64914680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:43:45.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:45.129+0000 7fcc64914680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:43:46.275 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:45.912+0000 7fcc64914680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:43:46.275 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:45.997+0000 7fcc64914680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:43:46.275 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:46.092+0000 7fcc64914680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:43:46.559 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:46.274+0000 7fcc64914680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:43:46.559 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:46.557+0000 7fcc64914680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:43:46.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:46.695+0000 7fcc64914680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:46.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:46.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:46 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: cluster 2024-02-10T09:43:45.685909+0000 mgr.y (mgr.34736) 24 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: audit 2024-02-10T09:43:46.232264+0000 mon.c (mon.1) 228 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: audit 2024-02-10T09:43:46.232905+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: audit 2024-02-10T09:43:46.236655+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: audit 2024-02-10T09:43:46.238954+0000 mon.c (mon.1) 229 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: audit 2024-02-10T09:43:46.239626+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:46.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:46 smithi204 bash[35809]: audit 2024-02-10T09:43:46.242850+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:47.134 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:46.857+0000 7fcc64914680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:43:47.134 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:47.042+0000 7fcc64914680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:43:47.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: cluster 2024-02-10T09:43:45.685909+0000 mgr.y (mgr.34736) 24 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:47.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: cluster 2024-02-10T09:43:45.685909+0000 mgr.y (mgr.34736) 24 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:47.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: audit 2024-02-10T09:43:46.232264+0000 mon.c (mon.1) 228 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:47.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: audit 2024-02-10T09:43:46.232264+0000 mon.c (mon.1) 228 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:47.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: audit 2024-02-10T09:43:46.232905+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:47.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: audit 2024-02-10T09:43:46.236655+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:47.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: audit 2024-02-10T09:43:46.238954+0000 mon.c (mon.1) 229 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:47.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: audit 2024-02-10T09:43:46.239626+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:47.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[35780]: audit 2024-02-10T09:43:46.242850+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:47.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: audit 2024-02-10T09:43:46.232905+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:47.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: audit 2024-02-10T09:43:46.236655+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_image_name\": \"prom/node-exporter\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230648Z\", \"created\": \"2024-02-10T09:26:14.659269Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:26:14.194108Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:43:46.231017Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:47.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: audit 2024-02-10T09:43:46.238954+0000 mon.c (mon.1) 229 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:47.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: audit 2024-02-10T09:43:46.239626+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:43:47.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:46 smithi153 bash[39117]: audit 2024-02-10T09:43:46.242850+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:43:47.254 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:43:47.132+0000 7fcc64914680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:43:47.254 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: [10/Feb/2024:09:43:47] ENGINE Bus STARTING 2024-02-10T09:43:47.254 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: CherryPy Checker: 2024-02-10T09:43:47.254 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: The Application mounted at '' has an empty config. 2024-02-10T09:43:47.254 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: 2024-02-10T09:43:47.255 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: [10/Feb/2024:09:43:47] ENGINE Serving on http://:::9283 2024-02-10T09:43:47.495 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: [10/Feb/2024:09:43:47] ENGINE Bus STARTED 2024-02-10T09:43:48.029 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: cluster 2024-02-10T09:43:46.238014+0000 mgr.y (mgr.34736) 25 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: cluster 2024-02-10T09:43:46.238238+0000 mgr.y (mgr.34736) 26 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: cephadm 2024-02-10T09:43:46.247456+0000 mgr.y (mgr.34736) 27 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: cephadm 2024-02-10T09:43:46.254810+0000 mgr.y (mgr.34736) 28 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: cluster 2024-02-10T09:43:47.147985+0000 mon.a (mon.0) 898 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: cluster 2024-02-10T09:43:47.148098+0000 mon.a (mon.0) 899 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: audit 2024-02-10T09:43:47.149208+0000 mon.b (mon.2) 392 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: audit 2024-02-10T09:43:47.149599+0000 mon.b (mon.2) 393 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: audit 2024-02-10T09:43:47.151758+0000 mon.b (mon.2) 394 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:43:48.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:47 smithi204 bash[35809]: audit 2024-02-10T09:43:47.152656+0000 mon.b (mon.2) 395 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: cluster 2024-02-10T09:43:46.238014+0000 mgr.y (mgr.34736) 25 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: cluster 2024-02-10T09:43:46.238238+0000 mgr.y (mgr.34736) 26 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: cephadm 2024-02-10T09:43:46.247456+0000 mgr.y (mgr.34736) 27 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: cephadm 2024-02-10T09:43:46.254810+0000 mgr.y (mgr.34736) 28 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:48.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: cluster 2024-02-10T09:43:47.147985+0000 mon.a (mon.0) 898 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: cluster 2024-02-10T09:43:47.148098+0000 mon.a (mon.0) 899 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: audit 2024-02-10T09:43:47.149208+0000 mon.b (mon.2) 392 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: audit 2024-02-10T09:43:47.149599+0000 mon.b (mon.2) 393 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: audit 2024-02-10T09:43:47.151758+0000 mon.b (mon.2) 394 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[35780]: audit 2024-02-10T09:43:47.152656+0000 mon.b (mon.2) 395 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: cluster 2024-02-10T09:43:46.238014+0000 mgr.y (mgr.34736) 25 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: cluster 2024-02-10T09:43:46.238238+0000 mgr.y (mgr.34736) 26 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: cephadm 2024-02-10T09:43:46.247456+0000 mgr.y (mgr.34736) 27 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: cephadm 2024-02-10T09:43:46.254810+0000 mgr.y (mgr.34736) 28 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: cluster 2024-02-10T09:43:47.147985+0000 mon.a (mon.0) 898 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: cluster 2024-02-10T09:43:47.148098+0000 mon.a (mon.0) 899 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:43:48.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: audit 2024-02-10T09:43:47.149208+0000 mon.b (mon.2) 392 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:43:48.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: audit 2024-02-10T09:43:47.149599+0000 mon.b (mon.2) 393 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:43:48.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: audit 2024-02-10T09:43:47.151758+0000 mon.b (mon.2) 394 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:43:48.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:47 smithi153 bash[39117]: audit 2024-02-10T09:43:47.152656+0000 mon.b (mon.2) 395 : audit [DBG] from='mgr.? 172.21.15.204:0/3964533624' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:43:48.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 bash[57125]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:48.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:48.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:48.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:48.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:48 smithi204 bash[35809]: cluster 2024-02-10T09:43:47.905913+0000 mon.a (mon.0) 900 : cluster [DBG] mgrmap e39: y(active, since 36s), standbys: x 2024-02-10T09:43:49.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:48 smithi153 bash[39117]: cluster 2024-02-10T09:43:47.905913+0000 mon.a (mon.0) 900 : cluster [DBG] mgrmap e39: y(active, since 36s), standbys: x 2024-02-10T09:43:49.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:48 smithi153 bash[35780]: cluster 2024-02-10T09:43:47.905913+0000 mon.a (mon.0) 900 : cluster [DBG] mgrmap e39: y(active, since 36s), standbys: x 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:43:49.126+0000 7f12c2f4e700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:49.190 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:49.191 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:49.192 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:49.193 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:43:49.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:49.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:49.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:48 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:49.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[57277]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:49.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:49 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:49.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:49 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:49.333 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:49 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:50.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: cluster 2024-02-10T09:43:48.238631+0000 mgr.y (mgr.34736) 29 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.129218+0000 mon.c (mon.1) 230 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.130474+0000 mon.c (mon.1) 231 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.130789+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.132440+0000 mon.c (mon.1) 232 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.133437+0000 mon.c (mon.1) 233 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.133693+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.140726+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.142597+0000 mon.c (mon.1) 234 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.143638+0000 mon.c (mon.1) 235 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.143952+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.150260+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]': finished 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.152863+0000 mon.c (mon.1) 236 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.153869+0000 mon.c (mon.1) 237 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.154181+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: cluster 2024-02-10T09:43:48.238631+0000 mgr.y (mgr.34736) 29 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.129218+0000 mon.c (mon.1) 230 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.130474+0000 mon.c (mon.1) 231 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.130789+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.132440+0000 mon.c (mon.1) 232 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.133437+0000 mon.c (mon.1) 233 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.133693+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.140726+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T09:43:50.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.142597+0000 mon.c (mon.1) 234 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.143638+0000 mon.c (mon.1) 235 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.143952+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.150260+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]': finished 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.152863+0000 mon.c (mon.1) 236 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.153869+0000 mon.c (mon.1) 237 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.154181+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.160658+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]': finished 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.162554+0000 mon.c (mon.1) 238 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.163493+0000 mon.c (mon.1) 239 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:43:50.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.163793+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.167641+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.168844+0000 mon.c (mon.1) 240 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.169759+0000 mon.c (mon.1) 241 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.170010+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.173797+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]': finished 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.175200+0000 mon.c (mon.1) 242 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.176058+0000 mon.c (mon.1) 243 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.176323+0000 mon.a (mon.0) 912 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.180289+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]': finished 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.181516+0000 mon.c (mon.1) 244 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.182318+0000 mon.c (mon.1) 245 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.182593+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:43:50.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[35780]: audit 2024-02-10T09:43:49.186596+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]': finished 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.160658+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]': finished 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.162554+0000 mon.c (mon.1) 238 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.163493+0000 mon.c (mon.1) 239 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.163793+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.167641+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.168844+0000 mon.c (mon.1) 240 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.169759+0000 mon.c (mon.1) 241 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.170010+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.173797+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]': finished 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.175200+0000 mon.c (mon.1) 242 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.176058+0000 mon.c (mon.1) 243 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.176323+0000 mon.a (mon.0) 912 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:43:50.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.180289+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]': finished 2024-02-10T09:43:50.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.181516+0000 mon.c (mon.1) 244 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.182318+0000 mon.c (mon.1) 245 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:43:50.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.182593+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:43:50.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:49 smithi153 bash[39117]: audit 2024-02-10T09:43:49.186596+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]': finished 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: cluster 2024-02-10T09:43:48.238631+0000 mgr.y (mgr.34736) 29 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.129218+0000 mon.c (mon.1) 230 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.130474+0000 mon.c (mon.1) 231 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.130789+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.132440+0000 mon.c (mon.1) 232 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.133437+0000 mon.c (mon.1) 233 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.133693+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.140726+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T09:43:50.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.142597+0000 mon.c (mon.1) 234 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.143638+0000 mon.c (mon.1) 235 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.143952+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.150260+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]': finished 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.152863+0000 mon.c (mon.1) 236 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.153869+0000 mon.c (mon.1) 237 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.154181+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.160658+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]': finished 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.162554+0000 mon.c (mon.1) 238 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.163493+0000 mon.c (mon.1) 239 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.163793+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.167641+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.168844+0000 mon.c (mon.1) 240 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.169759+0000 mon.c (mon.1) 241 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.170010+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.173797+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]': finished 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.175200+0000 mon.c (mon.1) 242 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.176058+0000 mon.c (mon.1) 243 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.176323+0000 mon.a (mon.0) 912 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.180289+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]': finished 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.181516+0000 mon.c (mon.1) 244 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.182318+0000 mon.c (mon.1) 245 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.182593+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:43:50.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:49 smithi204 bash[35809]: audit 2024-02-10T09:43:49.186596+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]': finished 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.127434+0000 mgr.y (mgr.34736) 30 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:51.144 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: main() 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:51.145 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.127434+0000 mgr.y (mgr.34736) 30 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:51.146 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: main() 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:51.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:43:51.148 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: main() 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.149 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.143347+0000 mgr.y (mgr.34736) 31 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.153617+0000 mgr.y (mgr.34736) 32 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.163276+0000 mgr.y (mgr.34736) 33 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.169528+0000 mgr.y (mgr.34736) 34 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.175834+0000 mgr.y (mgr.34736) 35 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[39117]: cephadm 2024-02-10T09:43:49.182125+0000 mgr.y (mgr.34736) 36 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: main() 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:43:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.143347+0000 mgr.y (mgr.34736) 31 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.153617+0000 mgr.y (mgr.34736) 32 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.163276+0000 mgr.y (mgr.34736) 33 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.169528+0000 mgr.y (mgr.34736) 34 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.175834+0000 mgr.y (mgr.34736) 35 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:43:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:50 smithi153 bash[35780]: cephadm 2024-02-10T09:43:49.182125+0000 mgr.y (mgr.34736) 36 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.127434+0000 mgr.y (mgr.34736) 30 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:51.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: main() 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:43:51.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: main() 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:43:51.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.143347+0000 mgr.y (mgr.34736) 31 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.153617+0000 mgr.y (mgr.34736) 32 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.163276+0000 mgr.y (mgr.34736) 33 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.169528+0000 mgr.y (mgr.34736) 34 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.175834+0000 mgr.y (mgr.34736) 35 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:43:51.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:50 smithi204 bash[35809]: cephadm 2024-02-10T09:43:49.182125+0000 mgr.y (mgr.34736) 36 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:43:51.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Succeeded. 2024-02-10T09:43:51.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:51.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Found left-over process 66121 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Found left-over process 66122 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: Starting Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Found left-over process 66121 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Found left-over process 66122 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:51.440 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:51.838 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[68311]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-10T09:43:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[39117]: cluster 2024-02-10T09:43:50.239405+0000 mgr.y (mgr.34736) 37 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[39117]: cephadm 2024-02-10T09:43:50.559438+0000 mgr.y (mgr.34736) 38 : cephadm [INF] Upgrade: Updating node-exporter.a (1/2) 2024-02-10T09:43:52.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[39117]: cephadm 2024-02-10T09:43:50.559584+0000 mgr.y (mgr.34736) 39 : cephadm [INF] Deploying daemon node-exporter.a on smithi153 2024-02-10T09:43:52.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[68311]: Getting image source signatures 2024-02-10T09:43:52.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[68311]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-10T09:43:52.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[68311]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-10T09:43:52.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[68311]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-10T09:43:52.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[35780]: cluster 2024-02-10T09:43:50.239405+0000 mgr.y (mgr.34736) 37 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:52.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[35780]: cephadm 2024-02-10T09:43:50.559438+0000 mgr.y (mgr.34736) 38 : cephadm [INF] Upgrade: Updating node-exporter.a (1/2) 2024-02-10T09:43:52.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:51 smithi153 bash[35780]: cephadm 2024-02-10T09:43:50.559584+0000 mgr.y (mgr.34736) 39 : cephadm [INF] Deploying daemon node-exporter.a on smithi153 2024-02-10T09:43:52.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:51 smithi204 bash[35809]: cluster 2024-02-10T09:43:50.239405+0000 mgr.y (mgr.34736) 37 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:52.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:51 smithi204 bash[35809]: cephadm 2024-02-10T09:43:50.559438+0000 mgr.y (mgr.34736) 38 : cephadm [INF] Upgrade: Updating node-exporter.a (1/2) 2024-02-10T09:43:52.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:51 smithi204 bash[35809]: cephadm 2024-02-10T09:43:50.559584+0000 mgr.y (mgr.34736) 39 : cephadm [INF] Deploying daemon node-exporter.a on smithi153 2024-02-10T09:43:52.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:52 smithi153 bash[68311]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-10T09:43:52.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:52 smithi153 bash[68311]: Writing manifest to image destination 2024-02-10T09:43:53.622 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.522Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T09:43:53.622 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.522Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T09:43:53.622 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.522Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T09:43:53.622 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.522Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T09:43:53.622 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T09:43:53.623 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T09:43:53.624 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T09:43:53.625 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T09:43:53.626 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a[68337]: ts=2024-02-10T09:43:53.523Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T09:43:53.626 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[68311]: 086dc26b8c2c983b7f38452ca4b2c40dfeb4ea67ae3c44059bd3481e8203006b 2024-02-10T09:43:53.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:43:53 smithi153 systemd[1]: Started Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:53.943 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:43:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:43:54.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:53 smithi204 bash[35809]: cluster 2024-02-10T09:43:52.239793+0000 mgr.y (mgr.34736) 40 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:54.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:53 smithi204 bash[35809]: audit 2024-02-10T09:43:53.668229+0000 mon.c (mon.1) 246 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:54.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:53 smithi204 bash[35809]: audit 2024-02-10T09:43:53.668810+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:54.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:53 smithi204 bash[35809]: audit 2024-02-10T09:43:53.672474+0000 mon.a (mon.0) 917 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:54.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[39117]: cluster 2024-02-10T09:43:52.239793+0000 mgr.y (mgr.34736) 40 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:54.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[35780]: cluster 2024-02-10T09:43:52.239793+0000 mgr.y (mgr.34736) 40 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:54.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[35780]: audit 2024-02-10T09:43:53.668229+0000 mon.c (mon.1) 246 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:54.414 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[39117]: audit 2024-02-10T09:43:53.668229+0000 mon.c (mon.1) 246 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:54.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[35780]: audit 2024-02-10T09:43:53.668810+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:54.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[35780]: audit 2024-02-10T09:43:53.672474+0000 mon.a (mon.0) 917 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:54.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[39117]: audit 2024-02-10T09:43:53.668810+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:54.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:53 smithi153 bash[39117]: audit 2024-02-10T09:43:53.672474+0000 mon.a (mon.0) 917 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 207618048, \"cpu_percentage\": \"1.23%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488723Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 432747315, \"cpu_percentage\": \"4.54%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:22.488911Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 194615705, \"cpu_percentage\": \"0.72%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.488985Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489063Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 46588231, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489123Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 45172654, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489182Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48307896, \"cpu_percentage\": \"0.53%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489242Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 37455134, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:22.489355Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\", \"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:43:22.489418Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:43:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:43:54] "GET /metrics HTTP/1.1" 200 153254 "" "Prometheus/2.33.4" 2024-02-10T09:43:55.946 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Succeeded. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Found left-over process 54416 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Found left-over process 54417 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: Starting Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Found left-over process 54416 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Found left-over process 54417 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:55.947 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:55 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:55 smithi204 bash[35809]: cluster 2024-02-10T09:43:54.240423+0000 mgr.y (mgr.34736) 41 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:56.331 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[57576]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-10T09:43:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:55 smithi153 bash[35780]: cluster 2024-02-10T09:43:54.240423+0000 mgr.y (mgr.34736) 41 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:56.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:55 smithi153 bash[39117]: cluster 2024-02-10T09:43:54.240423+0000 mgr.y (mgr.34736) 41 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:56.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[57576]: Getting image source signatures 2024-02-10T09:43:56.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[57576]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-10T09:43:56.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[57576]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-10T09:43:56.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[57576]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-10T09:43:57.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[35809]: cephadm 2024-02-10T09:43:55.083967+0000 mgr.y (mgr.34736) 42 : cephadm [INF] Upgrade: Updating node-exporter.b (2/2) 2024-02-10T09:43:57.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:56 smithi204 bash[35809]: cephadm 2024-02-10T09:43:55.084191+0000 mgr.y (mgr.34736) 43 : cephadm [INF] Deploying daemon node-exporter.b on smithi204 2024-02-10T09:43:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:56 smithi153 bash[35780]: cephadm 2024-02-10T09:43:55.083967+0000 mgr.y (mgr.34736) 42 : cephadm [INF] Upgrade: Updating node-exporter.b (2/2) 2024-02-10T09:43:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:56 smithi153 bash[35780]: cephadm 2024-02-10T09:43:55.084191+0000 mgr.y (mgr.34736) 43 : cephadm [INF] Deploying daemon node-exporter.b on smithi204 2024-02-10T09:43:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:56 smithi153 bash[39117]: cephadm 2024-02-10T09:43:55.083967+0000 mgr.y (mgr.34736) 42 : cephadm [INF] Upgrade: Updating node-exporter.b (2/2) 2024-02-10T09:43:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:56 smithi153 bash[39117]: cephadm 2024-02-10T09:43:55.084191+0000 mgr.y (mgr.34736) 43 : cephadm [INF] Deploying daemon node-exporter.b on smithi204 2024-02-10T09:43:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:57 smithi204 bash[35809]: cluster 2024-02-10T09:43:56.240847+0000 mgr.y (mgr.34736) 44 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:58.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:57 smithi204 bash[57576]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-10T09:43:58.332 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:57 smithi204 bash[57576]: Writing manifest to image destination 2024-02-10T09:43:58.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:57 smithi153 bash[35780]: cluster 2024-02-10T09:43:56.240847+0000 mgr.y (mgr.34736) 44 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:58.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:57 smithi153 bash[39117]: cluster 2024-02-10T09:43:56.240847+0000 mgr.y (mgr.34736) 44 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:58.965 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.638Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.638Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T09:43:58.966 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T09:43:58.967 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T09:43:58.968 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T09:43:58.969 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T09:43:58.969 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T09:43:58.969 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T09:43:58.969 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b[57627]: ts=2024-02-10T09:43:58.639Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T09:43:58.969 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[57576]: accc637b7e99f9cb74c2278f5c9d068640722928a1cde85bf037c17606abc4fc 2024-02-10T09:43:58.969 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:43:58 smithi204 systemd[1]: Started Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:59.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: cluster 2024-02-10T09:43:58.241374+0000 mgr.y (mgr.34736) 45 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:59.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: audit 2024-02-10T09:43:58.738154+0000 mon.c (mon.1) 247 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:59.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: audit 2024-02-10T09:43:58.738810+0000 mon.a (mon.0) 918 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:59.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: audit 2024-02-10T09:43:58.742629+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:59.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: audit 2024-02-10T09:43:58.743809+0000 mon.c (mon.1) 248 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:59.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: audit 2024-02-10T09:43:58.745539+0000 mon.c (mon.1) 249 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:59.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:43:58 smithi204 bash[35809]: audit 2024-02-10T09:43:58.745774+0000 mon.c (mon.1) 250 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:59.349 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: cluster 2024-02-10T09:43:58.241374+0000 mgr.y (mgr.34736) 45 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:59.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: cluster 2024-02-10T09:43:58.241374+0000 mgr.y (mgr.34736) 45 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:43:59.349 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: audit 2024-02-10T09:43:58.738154+0000 mon.c (mon.1) 247 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:59.350 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: audit 2024-02-10T09:43:58.738154+0000 mon.c (mon.1) 247 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:59.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: audit 2024-02-10T09:43:58.738810+0000 mon.a (mon.0) 918 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:59.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: audit 2024-02-10T09:43:58.742629+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:59.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: audit 2024-02-10T09:43:58.743809+0000 mon.c (mon.1) 248 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: audit 2024-02-10T09:43:58.745539+0000 mon.c (mon.1) 249 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[39117]: audit 2024-02-10T09:43:58.745774+0000 mon.c (mon.1) 250 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: audit 2024-02-10T09:43:58.738810+0000 mon.a (mon.0) 918 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: audit 2024-02-10T09:43:58.742629+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 198810009, \"cpu_percentage\": \"0.88%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230049Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 44721766, \"cpu_percentage\": \"107.48%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:43:46.230243Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230333Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230414Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230493Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47794094, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230571Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": true, \"last_refresh\": \"2024-02-10T09:43:46.230697Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 38839255, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:43:46.230791Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.28%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:43:46.230870Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: audit 2024-02-10T09:43:58.743809+0000 mon.c (mon.1) 248 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: audit 2024-02-10T09:43:58.745539+0000 mon.c (mon.1) 249 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:59.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:43:58 smithi153 bash[35780]: audit 2024-02-10T09:43:58.745774+0000 mon.c (mon.1) 250 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Service RestartSec=10s expired, scheduling restart. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:59.655 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 bash[57767]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:43:59.656 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:43:59 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:44:01.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:01 smithi204 bash[35809]: cluster 2024-02-10T09:44:00.241995+0000 mgr.y (mgr.34736) 46 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:01 smithi153 bash[35780]: cluster 2024-02-10T09:44:00.241995+0000 mgr.y (mgr.34736) 46 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:01 smithi153 bash[39117]: cluster 2024-02-10T09:44:00.241995+0000 mgr.y (mgr.34736) 46 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:02.939 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:02 smithi153 bash[66122]: time="2024-02-10T09:44:02Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:44:03.907 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:44:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:44:03.907 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[54417]: time="2024-02-10T09:44:03Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: cluster 2024-02-10T09:44:02.242548+0000 mgr.y (mgr.34736) 47 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:02.968044+0000 mon.c (mon.1) 251 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:02.968566+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:02.972003+0000 mon.a (mon.0) 921 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:03.598836+0000 mon.c (mon.1) 252 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:03.599425+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.165 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:03.603403+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:04.166 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:03.605585+0000 mon.c (mon.1) 253 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:04.166 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:03.606269+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:04.166 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:03 smithi204 bash[35809]: audit 2024-02-10T09:44:03.611630+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: cluster 2024-02-10T09:44:02.242548+0000 mgr.y (mgr.34736) 47 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:04.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: cluster 2024-02-10T09:44:02.242548+0000 mgr.y (mgr.34736) 47 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:04.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:02.968044+0000 mon.c (mon.1) 251 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:02.968044+0000 mon.c (mon.1) 251 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:02.968566+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:02.968566+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:02.972003+0000 mon.a (mon.0) 921 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:04.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:03.598836+0000 mon.c (mon.1) 252 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:02.972003+0000 mon.a (mon.0) 921 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"container_id\": \"f3366761db0a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 214853222, \"cpu_percentage\": \"1.25%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966065Z\", \"created\": \"2024-02-10T09:16:41.486607Z\", \"started\": \"2024-02-10T09:16:45.962801Z\"}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:52.727596Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:02.966849Z\", \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:04.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:03.599425+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:03.603403+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:04.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:03.605585+0000 mon.c (mon.1) 253 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:04.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:03.606269+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:04.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[35780]: audit 2024-02-10T09:44:03.611630+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:04.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:03.598836+0000 mon.c (mon.1) 252 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:03.599425+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:04.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:03.603403+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"72f83803a230\", \"container_image_id\": \"87a51ecf0b1c9a7b187b21c1b071425dafea0d765a96d5bc371c791169b3d7f4\", \"container_image_name\": \"docker.io/ceph/ceph-grafana:latest\", \"container_image_digests\": [\"docker.io/ceph/ceph-grafana@sha256:f5ec32c0cf2585fc34af7f6be4d829a1ba6fe9c8eac6f4593d62f258dd718a8a\"], \"memory_usage\": 37444648, \"cpu_percentage\": \"0.39%\", \"version\": \"6.6.2\", \"status\": -1, \"status_desc\": \"error\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597386Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:32:14.643472Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:28:43.346943Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:03.597653Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:04.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:03.605585+0000 mon.c (mon.1) 253 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:04.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:03.606269+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:04.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:03 smithi153 bash[39117]: audit 2024-02-10T09:44:03.611630+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:04] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:44:05.067 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:04 smithi204 bash[35809]: cluster 2024-02-10T09:44:03.604710+0000 mgr.y (mgr.34736) 48 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:05.068 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:04 smithi204 bash[35809]: cluster 2024-02-10T09:44:03.604958+0000 mgr.y (mgr.34736) 49 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:05.068 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:04 smithi204 bash[35809]: cephadm 2024-02-10T09:44:03.618926+0000 mgr.y (mgr.34736) 50 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:44:05.068 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:04 smithi204 bash[35809]: cephadm 2024-02-10T09:44:03.625390+0000 mgr.y (mgr.34736) 51 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:44:05.068 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:04 smithi204 bash[35809]: cluster 2024-02-10T09:44:04.601154+0000 mon.a (mon.0) 926 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:44:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[39117]: cluster 2024-02-10T09:44:03.604710+0000 mgr.y (mgr.34736) 48 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[39117]: cluster 2024-02-10T09:44:03.604958+0000 mgr.y (mgr.34736) 49 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[39117]: cephadm 2024-02-10T09:44:03.618926+0000 mgr.y (mgr.34736) 50 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:44:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[39117]: cephadm 2024-02-10T09:44:03.625390+0000 mgr.y (mgr.34736) 51 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:44:05.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[39117]: cluster 2024-02-10T09:44:04.601154+0000 mon.a (mon.0) 926 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:44:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[35780]: cluster 2024-02-10T09:44:03.604710+0000 mgr.y (mgr.34736) 48 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[35780]: cluster 2024-02-10T09:44:03.604958+0000 mgr.y (mgr.34736) 49 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[35780]: cephadm 2024-02-10T09:44:03.618926+0000 mgr.y (mgr.34736) 50 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-02-10T09:44:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[35780]: cephadm 2024-02-10T09:44:03.625390+0000 mgr.y (mgr.34736) 51 : cephadm [INF] Reconfiguring daemon grafana.a on smithi204 2024-02-10T09:44:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:04 smithi153 bash[35780]: cluster 2024-02-10T09:44:04.601154+0000 mon.a (mon.0) 926 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:44:06.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:06.582 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:44:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[58522]: Error: creating container storage: the container name "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" is already in use by 72f83803a230074fdca0a3059ad3ec7e059df3eeca56a83cb793aa72370fe5b7. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. 2024-02-10T09:44:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Control process exited, code=exited status=125 2024-02-10T09:44:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T09:44:06.583 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:06 smithi204 systemd[1]: Failed to start Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:44:06.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:44:06.399+0000 7f12c2f4e700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:06.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:06.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:06.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: yield (conn, connr) 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: code, '\n'.join(err))) 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Reconfig daemon grafana.a ... 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:06.691 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: Traceback (most recent call last): 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: main() 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: r = ctx.func(ctx) 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: return func(ctx) 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ports=daemon_ports) 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:06.692 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:06.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:06.693 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:44:07.150 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: cluster 2024-02-10T09:44:05.605725+0000 mgr.y (mgr.34736) 52 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:07.150 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.402874+0000 mon.c (mon.1) 254 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.404082+0000 mon.c (mon.1) 255 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.404405+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.406024+0000 mon.c (mon.1) 256 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.407585+0000 mon.c (mon.1) 257 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.409261+0000 mon.c (mon.1) 258 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.410828+0000 mon.c (mon.1) 259 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.412133+0000 mon.c (mon.1) 260 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.413289+0000 mon.c (mon.1) 261 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.414395+0000 mon.c (mon.1) 262 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.415806+0000 mon.c (mon.1) 263 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.417079+0000 mon.c (mon.1) 264 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.151 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:06 smithi204 bash[35809]: audit 2024-02-10T09:44:06.418348+0000 mon.c (mon.1) 265 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: cluster 2024-02-10T09:44:05.605725+0000 mgr.y (mgr.34736) 52 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.402874+0000 mon.c (mon.1) 254 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.404082+0000 mon.c (mon.1) 255 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.404405+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.406024+0000 mon.c (mon.1) 256 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.407585+0000 mon.c (mon.1) 257 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.409261+0000 mon.c (mon.1) 258 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.410828+0000 mon.c (mon.1) 259 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.412133+0000 mon.c (mon.1) 260 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.413289+0000 mon.c (mon.1) 261 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.414395+0000 mon.c (mon.1) 262 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.415806+0000 mon.c (mon.1) 263 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.417079+0000 mon.c (mon.1) 264 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[35780]: audit 2024-02-10T09:44:06.418348+0000 mon.c (mon.1) 265 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: cluster 2024-02-10T09:44:05.605725+0000 mgr.y (mgr.34736) 52 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.402874+0000 mon.c (mon.1) 254 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.404082+0000 mon.c (mon.1) 255 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.404405+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.406024+0000 mon.c (mon.1) 256 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.407585+0000 mon.c (mon.1) 257 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.409261+0000 mon.c (mon.1) 258 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.410828+0000 mon.c (mon.1) 259 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.412133+0000 mon.c (mon.1) 260 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.413289+0000 mon.c (mon.1) 261 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.414395+0000 mon.c (mon.1) 262 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.415806+0000 mon.c (mon.1) 263 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.417079+0000 mon.c (mon.1) 264 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:06 smithi153 bash[39117]: audit 2024-02-10T09:44:06.418348+0000 mon.c (mon.1) 265 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: cephadm 2024-02-10T09:44:06.401149+0000 mgr.y (mgr.34736) 53 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: main() 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: yield (conn, connr) 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: code, '\n'.join(err))) 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Reconfig daemon grafana.a ... 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: Traceback (most recent call last): 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: main() 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:08.084 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: r = ctx.func(ctx) 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: return func(ctx) 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: ports=daemon_ports) 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:07 smithi204 bash[35809]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: cephadm 2024-02-10T09:44:06.401149+0000 mgr.y (mgr.34736) 53 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: main() 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:08.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: yield (conn, connr) 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: code, '\n'.join(err))) 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Reconfig daemon grafana.a ... 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: cephadm 2024-02-10T09:44:06.401149+0000 mgr.y (mgr.34736) 53 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: main() 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:08.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: yield (conn, connr) 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2024-02-10T09:44:08.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: code, '\n'.join(err))) 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: /bin/podman: stderr Error: no such container ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Reconfig daemon grafana.a ... 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: Traceback (most recent call last): 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: main() 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: r = ctx.func(ctx) 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: return func(ctx) 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:08.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: ports=daemon_ports) 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[39117]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Non-zero exit code 1 from systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: systemctl: stderr Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: systemctl: stderr See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: Traceback (most recent call last): 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-02-10T09:44:08.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: main() 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: r = ctx.func(ctx) 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2061, in _default_image 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: return func(ctx) 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5293, in command_deploy 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: ports=daemon_ports) 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 3151, in deploy_daemon 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: File "/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1719, in call_throws 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: RuntimeError: Failed command: systemctl restart ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a: Job for ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service failed because the control process exited with error code. 2024-02-10T09:44:08.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:07 smithi153 bash[35780]: See "systemctl status ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service" and "journalctl -xe" for details. 2024-02-10T09:44:09.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:08 smithi204 bash[35809]: cluster 2024-02-10T09:44:07.606258+0000 mgr.y (mgr.34736) 54 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:09.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:08 smithi204 bash[35809]: cephadm 2024-02-10T09:44:07.823097+0000 mgr.y (mgr.34736) 55 : cephadm [INF] Upgrade: Updating grafana.a 2024-02-10T09:44:09.137 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:08 smithi204 bash[35809]: cephadm 2024-02-10T09:44:07.829719+0000 mgr.y (mgr.34736) 56 : cephadm [INF] Deploying daemon grafana.a on smithi204 2024-02-10T09:44:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:08 smithi153 bash[35780]: cluster 2024-02-10T09:44:07.606258+0000 mgr.y (mgr.34736) 54 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:08 smithi153 bash[35780]: cephadm 2024-02-10T09:44:07.823097+0000 mgr.y (mgr.34736) 55 : cephadm [INF] Upgrade: Updating grafana.a 2024-02-10T09:44:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:08 smithi153 bash[35780]: cephadm 2024-02-10T09:44:07.829719+0000 mgr.y (mgr.34736) 56 : cephadm [INF] Deploying daemon grafana.a on smithi204 2024-02-10T09:44:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:08 smithi153 bash[39117]: cluster 2024-02-10T09:44:07.606258+0000 mgr.y (mgr.34736) 54 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:08 smithi153 bash[39117]: cephadm 2024-02-10T09:44:07.823097+0000 mgr.y (mgr.34736) 55 : cephadm [INF] Upgrade: Updating grafana.a 2024-02-10T09:44:09.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:08 smithi153 bash[39117]: cephadm 2024-02-10T09:44:07.829719+0000 mgr.y (mgr.34736) 56 : cephadm [INF] Deploying daemon grafana.a on smithi204 2024-02-10T09:44:10.519 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: Starting Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49569 (bash) in control group while starting unit. Ignoring. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49570 (podman) in control group while starting unit. Ignoring. 2024-02-10T09:44:10.832 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Found left-over process 49586 (conmon) in control group while starting unit. Ignoring. 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped Stream Manager" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2024-02-10T09:44:10.833 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[49570]: t=2024-02-10T09:44:10+0000 lvl=eror msg="Server shutdown" logger=server reason=nil 2024-02-10T09:44:11.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:10 smithi153 bash[35780]: cluster 2024-02-10T09:44:09.607140+0000 mgr.y (mgr.34736) 57 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:11.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:10 smithi153 bash[39117]: cluster 2024-02-10T09:44:09.607140+0000 mgr.y (mgr.34736) 57 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:11.258 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:10 smithi204 bash[35809]: cluster 2024-02-10T09:44:09.607140+0000 mgr.y (mgr.34736) 57 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:11.581 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 bash[58877]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T09:44:11.840 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-10T09:44:11.841 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-10T09:44:11.842 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-10T09:44:11.843 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-10T09:44:11.844 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-10T09:44:11.844 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-10T09:44:11.844 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-10T09:44:11.844 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T09:44:12.093 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 bash[58953]: 67a3139af0c067627de43c15be661657896f4a076b4c9b8f1c88fb8d04d41394 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 systemd[1]: Started Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-10T09:44:12.094 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T09:44:12.095 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-10T09:44:12.096 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-10T09:44:12.097 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-10T09:44:12.098 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T09:44:12.099 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-10T09:44:12.100 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-10T09:44:12.101 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-10T09:44:12.102 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-10T09:44:12.103 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-10T09:44:12.104 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-10T09:44:12.105 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-10T09:44:12.106 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-10T09:44:12.398 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-10T09:44:12.398 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-10T09:44:12.398 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-10T09:44:12.398 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-10T09:44:12.398 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T09:44:12.398 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-10T09:44:12.399 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-10T09:44:12.400 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-10T09:44:12.401 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T09:44:12.402 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T09:44:12.403 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-10T09:44:12.404 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-10T09:44:12.405 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-10T09:44:12.406 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-10T09:44:12.407 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=454.581191ms 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-10T09:44:12.408 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T09:44:12.714 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-10T09:44:12.714 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T09:44:12.714 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:44:12 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:44:12+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-10T09:44:12.783 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:44:13.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:12 smithi204 bash[35809]: cluster 2024-02-10T09:44:11.607585+0000 mgr.y (mgr.34736) 58 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:13.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:12 smithi204 bash[35809]: audit 2024-02-10T09:44:11.922108+0000 mon.c (mon.1) 266 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:13.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:12 smithi204 bash[35809]: audit 2024-02-10T09:44:11.922694+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:13.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:12 smithi204 bash[35809]: audit 2024-02-10T09:44:11.928353+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:13.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:12 smithi204 bash[35809]: audit 2024-02-10T09:44:11.931490+0000 mon.c (mon.1) 267 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:13.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:12 smithi204 bash[35809]: audit 2024-02-10T09:44:11.938321+0000 mon.c (mon.1) 268 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:44:13.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[39117]: cluster 2024-02-10T09:44:11.607585+0000 mgr.y (mgr.34736) 58 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:13.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[35780]: cluster 2024-02-10T09:44:11.607585+0000 mgr.y (mgr.34736) 58 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:13.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[35780]: audit 2024-02-10T09:44:11.922108+0000 mon.c (mon.1) 266 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:13.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[39117]: audit 2024-02-10T09:44:11.922108+0000 mon.c (mon.1) 266 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[39117]: audit 2024-02-10T09:44:11.922694+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[35780]: audit 2024-02-10T09:44:11.922694+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[35780]: audit 2024-02-10T09:44:11.928353+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[35780]: audit 2024-02-10T09:44:11.931490+0000 mon.c (mon.1) 267 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[35780]: audit 2024-02-10T09:44:11.938321+0000 mon.c (mon.1) 268 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[39117]: audit 2024-02-10T09:44:11.928353+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.89%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.596784Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 403911475, \"cpu_percentage\": \"36.73%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:03.596945Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50121932, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597028Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597101Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48790241, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597173Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597243Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 3465543, \"cpu_percentage\": \"10.87%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:03.597314Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38912655, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:03.597458Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:03.597528Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[39117]: audit 2024-02-10T09:44:11.931490+0000 mon.c (mon.1) 267 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:13.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:12 smithi153 bash[39117]: audit 2024-02-10T09:44:11.938321+0000 mon.c (mon.1) 268 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (12m) 10s ago 13m 26.0M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 starting - - - - 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (32s) 9s ago 25m 385M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (8m) 10s ago 27m 427M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (27m) 10s ago 27m 204M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:44:13.388 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (25m) 9s ago 25m 189M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (26m) 10s ago 26m 185M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (20s) 10s ago 17m 11.9M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (15s) 9s ago 17m 3384k - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (24m) 10s ago 25m 43.8M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (24m) 10s ago 25m 44.4M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (24m) 10s ago 24m 43.1M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (24m) 10s ago 24m 46.0M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (24m) 9s ago 24m 47.7M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (23m) 9s ago 23m 48.7M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (23m) 9s ago 23m 46.5M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (23m) 9s ago 23m 45.5M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (12m) 9s ago 12m 66.0M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:44:13.389 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (15m) 10s ago 15m 35.9M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:44:13.390 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (15m) 9s ago 15m 37.1M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:44:13.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:44:13.757 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:44:13.913 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:44:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:44:14.065 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:44:14.065 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:44:14.065 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:44:14.065 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T09:44:14.065 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-02-10T09:44:14.066 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-02-10T09:44:14.066 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-02-10T09:44:14.066 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "2/2 daemons upgraded", 2024-02-10T09:44:14.066 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading grafana daemons", 2024-02-10T09:44:14.066 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:44:14.066 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:12.771508+0000 mgr.y (mgr.34736) 59 : audit [DBG] from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.378855+0000 mon.c (mon.1) 269 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.379249+0000 mon.c (mon.1) 270 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.379531+0000 mon.c (mon.1) 271 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.379835+0000 mon.c (mon.1) 272 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.380113+0000 mon.c (mon.1) 273 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.380390+0000 mon.c (mon.1) 274 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.381866+0000 mon.c (mon.1) 275 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.382144+0000 mon.c (mon.1) 276 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.382435+0000 mon.c (mon.1) 277 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.382707+0000 mon.c (mon.1) 278 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.382985+0000 mon.c (mon.1) 279 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[39117]: audit 2024-02-10T09:44:13.755517+0000 mon.a (mon.0) 930 : audit [DBG] from='client.? 172.21.15.153:0/1359735975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:12.771508+0000 mgr.y (mgr.34736) 59 : audit [DBG] from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.378855+0000 mon.c (mon.1) 269 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.379249+0000 mon.c (mon.1) 270 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.379531+0000 mon.c (mon.1) 271 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.379835+0000 mon.c (mon.1) 272 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.380113+0000 mon.c (mon.1) 273 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.380390+0000 mon.c (mon.1) 274 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.381866+0000 mon.c (mon.1) 275 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.382144+0000 mon.c (mon.1) 276 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.382435+0000 mon.c (mon.1) 277 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.382707+0000 mon.c (mon.1) 278 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.382985+0000 mon.c (mon.1) 279 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:13 smithi153 bash[35780]: audit 2024-02-10T09:44:13.755517+0000 mon.a (mon.0) 930 : audit [DBG] from='client.? 172.21.15.153:0/1359735975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:12.771508+0000 mgr.y (mgr.34736) 59 : audit [DBG] from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:14.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.378855+0000 mon.c (mon.1) 269 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.379249+0000 mon.c (mon.1) 270 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.379531+0000 mon.c (mon.1) 271 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.379835+0000 mon.c (mon.1) 272 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.380113+0000 mon.c (mon.1) 273 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.380390+0000 mon.c (mon.1) 274 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.381866+0000 mon.c (mon.1) 275 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.382144+0000 mon.c (mon.1) 276 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.382435+0000 mon.c (mon.1) 277 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.382707+0000 mon.c (mon.1) 278 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.382985+0000 mon.c (mon.1) 279 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:44:14.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:13 smithi204 bash[35809]: audit 2024-02-10T09:44:13.755517+0000 mon.a (mon.0) 930 : audit [DBG] from='client.? 172.21.15.153:0/1359735975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:14.915 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:14] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:44:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:14 smithi153 bash[35780]: audit 2024-02-10T09:44:13.080934+0000 mgr.y (mgr.34736) 60 : audit [DBG] from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:14 smithi153 bash[35780]: audit 2024-02-10T09:44:13.377002+0000 mgr.y (mgr.34736) 61 : audit [DBG] from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:15.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:14 smithi153 bash[35780]: cluster 2024-02-10T09:44:13.607965+0000 mgr.y (mgr.34736) 62 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:15.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:14 smithi153 bash[39117]: audit 2024-02-10T09:44:13.080934+0000 mgr.y (mgr.34736) 60 : audit [DBG] from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:14 smithi153 bash[39117]: audit 2024-02-10T09:44:13.377002+0000 mgr.y (mgr.34736) 61 : audit [DBG] from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:15.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:14 smithi153 bash[39117]: cluster 2024-02-10T09:44:13.607965+0000 mgr.y (mgr.34736) 62 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:14 smithi204 bash[35809]: audit 2024-02-10T09:44:13.080934+0000 mgr.y (mgr.34736) 60 : audit [DBG] from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:14 smithi204 bash[35809]: audit 2024-02-10T09:44:13.377002+0000 mgr.y (mgr.34736) 61 : audit [DBG] from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:15.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:14 smithi204 bash[35809]: cluster 2024-02-10T09:44:13.607965+0000 mgr.y (mgr.34736) 62 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:16.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:15 smithi153 bash[35780]: audit 2024-02-10T09:44:14.064986+0000 mgr.y (mgr.34736) 63 : audit [DBG] from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:16.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:15 smithi153 bash[39117]: audit 2024-02-10T09:44:14.064986+0000 mgr.y (mgr.34736) 63 : audit [DBG] from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:16.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:15 smithi204 bash[35809]: audit 2024-02-10T09:44:14.064986+0000 mgr.y (mgr.34736) 63 : audit [DBG] from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:17.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:44:16] ENGINE Bus STOPPING 2024-02-10T09:44:17.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:44:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:44:17.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:44:16] ENGINE Bus STOPPED 2024-02-10T09:44:17.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:44:16] ENGINE Bus STARTING 2024-02-10T09:44:17.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:44:17] ENGINE Serving on http://:::9283 2024-02-10T09:44:17.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:44:17] ENGINE Bus STARTED 2024-02-10T09:44:17.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: cluster 2024-02-10T09:44:15.608615+0000 mgr.y (mgr.34736) 64 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:17.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: cluster 2024-02-10T09:44:15.608615+0000 mgr.y (mgr.34736) 64 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:17.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.848790+0000 mon.c (mon.1) 280 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:17.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.848790+0000 mon.c (mon.1) 280 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:17.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.849440+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:17.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.849440+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:17.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.853652+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:17.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.855830+0000 mon.c (mon.1) 281 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:17.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.856133+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:17.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.861477+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.869317+0000 mon.c (mon.1) 282 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.870694+0000 mon.c (mon.1) 283 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.871394+0000 mon.c (mon.1) 284 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]: dispatch 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.853652+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.855830+0000 mon.c (mon.1) 281 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.856133+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.861477+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:17.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.869317+0000 mon.c (mon.1) 282 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.870694+0000 mon.c (mon.1) 283 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.871394+0000 mon.c (mon.1) 284 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.871700+0000 mon.a (mon.0) 935 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.878772+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]': finished 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.881727+0000 mon.c (mon.1) 285 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.882919+0000 mon.c (mon.1) 286 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.883266+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:17.194 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.885623+0000 mon.c (mon.1) 287 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.887718+0000 mon.c (mon.1) 288 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.890772+0000 mon.c (mon.1) 289 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.892696+0000 mon.c (mon.1) 290 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.894004+0000 mon.c (mon.1) 291 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.895266+0000 mon.c (mon.1) 292 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.896507+0000 mon.c (mon.1) 293 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.898067+0000 mon.c (mon.1) 294 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.899404+0000 mon.c (mon.1) 295 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.900863+0000 mon.c (mon.1) 296 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.902685+0000 mon.c (mon.1) 297 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.908667+0000 mon.c (mon.1) 298 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.908938+0000 mon.a (mon.0) 938 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.915159+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]': finished 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.916350+0000 mon.c (mon.1) 299 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:44:17.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[35780]: audit 2024-02-10T09:44:16.916651+0000 mon.a (mon.0) 940 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.871700+0000 mon.a (mon.0) 935 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.878772+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]': finished 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.881727+0000 mon.c (mon.1) 285 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.882919+0000 mon.c (mon.1) 286 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.883266+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.885623+0000 mon.c (mon.1) 287 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.887718+0000 mon.c (mon.1) 288 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.890772+0000 mon.c (mon.1) 289 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.892696+0000 mon.c (mon.1) 290 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.894004+0000 mon.c (mon.1) 291 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.895266+0000 mon.c (mon.1) 292 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.896507+0000 mon.c (mon.1) 293 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.898067+0000 mon.c (mon.1) 294 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.899404+0000 mon.c (mon.1) 295 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.900863+0000 mon.c (mon.1) 296 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.902685+0000 mon.c (mon.1) 297 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.908667+0000 mon.c (mon.1) 298 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.908938+0000 mon.a (mon.0) 938 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.915159+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]': finished 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.916350+0000 mon.c (mon.1) 299 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:44:17.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:16 smithi153 bash[39117]: audit 2024-02-10T09:44:16.916651+0000 mon.a (mon.0) 940 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:44:17.233 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: cluster 2024-02-10T09:44:15.608615+0000 mgr.y (mgr.34736) 64 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.848790+0000 mon.c (mon.1) 280 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.849440+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.853652+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"c3b5b5ecd32c\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 198914867, \"cpu_percentage\": \"0.90%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846285Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:18:21.578355Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:32:00.420097Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:44:16.847327Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.855830+0000 mon.c (mon.1) 281 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.856133+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.861477+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.869317+0000 mon.c (mon.1) 282 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.870694+0000 mon.c (mon.1) 283 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.871394+0000 mon.c (mon.1) 284 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]: dispatch 2024-02-10T09:44:17.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.871700+0000 mon.a (mon.0) 935 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.878772+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config set","value":"https://smithi204.front.sepia.ceph.com:3000","who":"mgr","name":"mgr/dashboard/GRAFANA_API_URL"}]': finished 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.881727+0000 mon.c (mon.1) 285 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.882919+0000 mon.c (mon.1) 286 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.883266+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.885623+0000 mon.c (mon.1) 287 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.887718+0000 mon.c (mon.1) 288 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.890772+0000 mon.c (mon.1) 289 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.892696+0000 mon.c (mon.1) 290 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.894004+0000 mon.c (mon.1) 291 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.895266+0000 mon.c (mon.1) 292 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.235 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.896507+0000 mon.c (mon.1) 293 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.898067+0000 mon.c (mon.1) 294 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.899404+0000 mon.c (mon.1) 295 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.900863+0000 mon.c (mon.1) 296 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.902685+0000 mon.c (mon.1) 297 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.908667+0000 mon.c (mon.1) 298 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.908938+0000 mon.a (mon.0) 938 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.915159+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]': finished 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.916350+0000 mon.c (mon.1) 299 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:44:17.236 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:16 smithi204 bash[35809]: audit 2024-02-10T09:44:16.916651+0000 mon.a (mon.0) 940 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:44:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: cluster 2024-02-10T09:44:16.855059+0000 mgr.y (mgr.34736) 65 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.869813+0000 mgr.y (mgr.34736) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T09:44:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.871075+0000 mgr.y (mgr.34736) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T09:44:18.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: cephadm 2024-02-10T09:44:16.908414+0000 mgr.y (mgr.34736) 68 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.922185+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.923043+0000 mon.c (mon.1) 300 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.923564+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.934743+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.936134+0000 mon.c (mon.1) 301 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.936382+0000 mon.a (mon.0) 944 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.939897+0000 mon.a (mon.0) 945 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.941629+0000 mon.c (mon.1) 302 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.941880+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.942501+0000 mon.c (mon.1) 303 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.942713+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.946418+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:44:18.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.947779+0000 mon.c (mon.1) 304 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: cluster 2024-02-10T09:44:16.855059+0000 mgr.y (mgr.34736) 65 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.869813+0000 mgr.y (mgr.34736) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.871075+0000 mgr.y (mgr.34736) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: cephadm 2024-02-10T09:44:16.908414+0000 mgr.y (mgr.34736) 68 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.922185+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.923043+0000 mon.c (mon.1) 300 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.923564+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.934743+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.936134+0000 mon.c (mon.1) 301 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.936382+0000 mon.a (mon.0) 944 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.939897+0000 mon.a (mon.0) 945 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.941629+0000 mon.c (mon.1) 302 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:44:18.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.941880+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.942501+0000 mon.c (mon.1) 303 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.942713+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.946418+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.947779+0000 mon.c (mon.1) 304 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.948039+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.948655+0000 mon.c (mon.1) 305 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.948904+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.952616+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.954271+0000 mon.c (mon.1) 306 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.954505+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.955117+0000 mon.c (mon.1) 307 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.955336+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.958995+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.960579+0000 mon.c (mon.1) 308 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.960802+0000 mon.a (mon.0) 955 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.964308+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.965825+0000 mon.c (mon.1) 309 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.966093+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.966723+0000 mon.c (mon.1) 310 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.966954+0000 mon.a (mon.0) 958 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.967544+0000 mon.c (mon.1) 311 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.967763+0000 mon.a (mon.0) 959 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.948039+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.948655+0000 mon.c (mon.1) 305 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.948904+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.952616+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.954271+0000 mon.c (mon.1) 306 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.954505+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.955117+0000 mon.c (mon.1) 307 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.955336+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.958995+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.960579+0000 mon.c (mon.1) 308 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.960802+0000 mon.a (mon.0) 955 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.964308+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.965825+0000 mon.c (mon.1) 309 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.966093+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.966723+0000 mon.c (mon.1) 310 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.966954+0000 mon.a (mon.0) 958 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.967544+0000 mon.c (mon.1) 311 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.967763+0000 mon.a (mon.0) 959 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.968344+0000 mon.c (mon.1) 312 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.968564+0000 mon.a (mon.0) 960 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.969151+0000 mon.c (mon.1) 313 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.969365+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.972892+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.974679+0000 mon.c (mon.1) 314 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.974933+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.977614+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:44:18.195 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.978906+0000 mon.c (mon.1) 315 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.980710+0000 mon.c (mon.1) 316 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.980968+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.983878+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.989042+0000 mon.c (mon.1) 317 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.990767+0000 mon.c (mon.1) 318 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.991009+0000 mon.a (mon.0) 967 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: audit 2024-02-10T09:44:16.994364+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: cluster 2024-02-10T09:44:17.851091+0000 mon.a (mon.0) 969 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[35780]: cluster 2024-02-10T09:44:17.851156+0000 mon.a (mon.0) 970 : cluster [INF] Cluster is now healthy 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.968344+0000 mon.c (mon.1) 312 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.968564+0000 mon.a (mon.0) 960 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.969151+0000 mon.c (mon.1) 313 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.969365+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.972892+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.974679+0000 mon.c (mon.1) 314 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.974933+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.977614+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.978906+0000 mon.c (mon.1) 315 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.980710+0000 mon.c (mon.1) 316 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.980968+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.983878+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.989042+0000 mon.c (mon.1) 317 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.990767+0000 mon.c (mon.1) 318 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.991009+0000 mon.a (mon.0) 967 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: audit 2024-02-10T09:44:16.994364+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: cluster 2024-02-10T09:44:17.851091+0000 mon.a (mon.0) 969 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-10T09:44:18.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:17 smithi153 bash[39117]: cluster 2024-02-10T09:44:17.851156+0000 mon.a (mon.0) 970 : cluster [INF] Cluster is now healthy 2024-02-10T09:44:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: cluster 2024-02-10T09:44:16.855059+0000 mgr.y (mgr.34736) 65 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.869813+0000 mgr.y (mgr.34736) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T09:44:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.871075+0000 mgr.y (mgr.34736) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T09:44:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: cephadm 2024-02-10T09:44:16.908414+0000 mgr.y (mgr.34736) 68 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:44:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.922185+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:44:18.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.923043+0000 mon.c (mon.1) 300 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.923564+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.934743+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.936134+0000 mon.c (mon.1) 301 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.936382+0000 mon.a (mon.0) 944 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.939897+0000 mon.a (mon.0) 945 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.941629+0000 mon.c (mon.1) 302 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.941880+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.942501+0000 mon.c (mon.1) 303 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.942713+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.946418+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.947779+0000 mon.c (mon.1) 304 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.948039+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.948655+0000 mon.c (mon.1) 305 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.948904+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:44:18.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.952616+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.954271+0000 mon.c (mon.1) 306 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.954505+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.955117+0000 mon.c (mon.1) 307 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.955336+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.958995+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.960579+0000 mon.c (mon.1) 308 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.960802+0000 mon.a (mon.0) 955 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.964308+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.965825+0000 mon.c (mon.1) 309 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.966093+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.966723+0000 mon.c (mon.1) 310 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.966954+0000 mon.a (mon.0) 958 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.967544+0000 mon.c (mon.1) 311 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.967763+0000 mon.a (mon.0) 959 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.968344+0000 mon.c (mon.1) 312 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.968564+0000 mon.a (mon.0) 960 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.969151+0000 mon.c (mon.1) 313 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.969365+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.972892+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.974679+0000 mon.c (mon.1) 314 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.974933+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.977614+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.978906+0000 mon.c (mon.1) 315 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.980710+0000 mon.c (mon.1) 316 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.980968+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.983878+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.989042+0000 mon.c (mon.1) 317 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.990767+0000 mon.c (mon.1) 318 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.991009+0000 mon.a (mon.0) 967 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:44:18.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: audit 2024-02-10T09:44:16.994364+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:44:18.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: cluster 2024-02-10T09:44:17.851091+0000 mon.a (mon.0) 969 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-10T09:44:18.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:17 smithi204 bash[35809]: cluster 2024-02-10T09:44:17.851156+0000 mon.a (mon.0) 970 : cluster [INF] Cluster is now healthy 2024-02-10T09:44:19.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: cephadm 2024-02-10T09:44:16.974271+0000 mgr.y (mgr.34736) 69 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:44:19.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.597194+0000 mon.c (mon.1) 319 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:44:19.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.597412+0000 mon.c (mon.1) 320 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.597654+0000 mon.c (mon.1) 321 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.597656+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.597926+0000 mon.c (mon.1) 322 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.598267+0000 mon.a (mon.0) 972 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.598660+0000 mon.a (mon.0) 973 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.598949+0000 mon.a (mon.0) 974 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.736169+0000 mon.c (mon.1) 323 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.736520+0000 mon.a (mon.0) 975 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.740517+0000 mon.a (mon.0) 976 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]': finished 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: cephadm 2024-02-10T09:44:16.974271+0000 mgr.y (mgr.34736) 69 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.597194+0000 mon.c (mon.1) 319 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:44:19.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.597412+0000 mon.c (mon.1) 320 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.597654+0000 mon.c (mon.1) 321 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.597656+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.597926+0000 mon.c (mon.1) 322 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.598267+0000 mon.a (mon.0) 972 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.598660+0000 mon.a (mon.0) 973 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.598949+0000 mon.a (mon.0) 974 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.736169+0000 mon.c (mon.1) 323 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.736520+0000 mon.a (mon.0) 975 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.740517+0000 mon.a (mon.0) 976 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]': finished 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.768579+0000 mon.c (mon.1) 324 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.768990+0000 mon.a (mon.0) 977 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:44:19.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[35780]: audit 2024-02-10T09:44:18.774350+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:44:19.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.768579+0000 mon.c (mon.1) 324 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:44:19.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.768990+0000 mon.a (mon.0) 977 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:44:19.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:18 smithi153 bash[39117]: audit 2024-02-10T09:44:18.774350+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: cephadm 2024-02-10T09:44:16.974271+0000 mgr.y (mgr.34736) 69 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.597194+0000 mon.c (mon.1) 319 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.597412+0000 mon.c (mon.1) 320 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.597654+0000 mon.c (mon.1) 321 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.597656+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.597926+0000 mon.c (mon.1) 322 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.598267+0000 mon.a (mon.0) 972 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.598660+0000 mon.a (mon.0) 973 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.598949+0000 mon.a (mon.0) 974 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.736169+0000 mon.c (mon.1) 323 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]: dispatch 2024-02-10T09:44:19.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.736520+0000 mon.a (mon.0) 975 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]: dispatch 2024-02-10T09:44:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.740517+0000 mon.a (mon.0) 976 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/prometheus/health_history","val":"{\"CEPHADM_FAILED_DAEMON\": {\"name\": \"CEPHADM_FAILED_DAEMON\", \"severity\": \"HEALTH_WARN\", \"first_seen\": 1707557497.0242572, \"last_seen\": 1707557497.0242572, \"count\": 1, \"active\": false}}"}]': finished 2024-02-10T09:44:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.768579+0000 mon.c (mon.1) 324 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:44:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.768990+0000 mon.a (mon.0) 977 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:44:19.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:18 smithi204 bash[35809]: audit 2024-02-10T09:44:18.774350+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:44:20.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:19 smithi153 bash[35780]: cluster 2024-02-10T09:44:18.855449+0000 mgr.y (mgr.34736) 70 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:20.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:19 smithi153 bash[39117]: cluster 2024-02-10T09:44:18.855449+0000 mgr.y (mgr.34736) 70 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:20.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:19 smithi204 bash[35809]: cluster 2024-02-10T09:44:18.855449+0000 mgr.y (mgr.34736) 70 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:22.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:21 smithi204 bash[35809]: cluster 2024-02-10T09:44:20.856224+0000 mgr.y (mgr.34736) 71 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:22.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:21 smithi153 bash[35780]: cluster 2024-02-10T09:44:20.856224+0000 mgr.y (mgr.34736) 71 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:22.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:21 smithi153 bash[39117]: cluster 2024-02-10T09:44:20.856224+0000 mgr.y (mgr.34736) 71 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:23.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:22 smithi204 bash[35809]: cluster 2024-02-10T09:44:22.856834+0000 mgr.y (mgr.34736) 72 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:23.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:22 smithi153 bash[35780]: cluster 2024-02-10T09:44:22.856834+0000 mgr.y (mgr.34736) 72 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:23.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:22 smithi153 bash[39117]: cluster 2024-02-10T09:44:22.856834+0000 mgr.y (mgr.34736) 72 : cluster [DBG] pgmap v39: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:44:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:44:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:24] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:44:26.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:25 smithi153 bash[35780]: cluster 2024-02-10T09:44:24.857649+0000 mgr.y (mgr.34736) 73 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:26.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:25 smithi153 bash[39117]: cluster 2024-02-10T09:44:24.857649+0000 mgr.y (mgr.34736) 73 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:26.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:25 smithi204 bash[35809]: cluster 2024-02-10T09:44:24.857649+0000 mgr.y (mgr.34736) 73 : cluster [DBG] pgmap v40: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:28.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:27 smithi153 bash[35780]: cluster 2024-02-10T09:44:26.858130+0000 mgr.y (mgr.34736) 74 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:28.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:27 smithi153 bash[39117]: cluster 2024-02-10T09:44:26.858130+0000 mgr.y (mgr.34736) 74 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:28.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:27 smithi204 bash[35809]: cluster 2024-02-10T09:44:26.858130+0000 mgr.y (mgr.34736) 74 : cluster [DBG] pgmap v41: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:30.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:29 smithi153 bash[35780]: cluster 2024-02-10T09:44:28.858696+0000 mgr.y (mgr.34736) 75 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:30.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:29 smithi153 bash[39117]: cluster 2024-02-10T09:44:28.858696+0000 mgr.y (mgr.34736) 75 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:30.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:29 smithi204 bash[35809]: cluster 2024-02-10T09:44:28.858696+0000 mgr.y (mgr.34736) 75 : cluster [DBG] pgmap v42: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:32.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:31 smithi153 bash[35780]: cluster 2024-02-10T09:44:30.859400+0000 mgr.y (mgr.34736) 76 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:32.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:31 smithi153 bash[39117]: cluster 2024-02-10T09:44:30.859400+0000 mgr.y (mgr.34736) 76 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:32.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:31 smithi204 bash[35809]: cluster 2024-02-10T09:44:30.859400+0000 mgr.y (mgr.34736) 76 : cluster [DBG] pgmap v43: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:33.937 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:44:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:44:34.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:33 smithi153 bash[35780]: cluster 2024-02-10T09:44:32.859944+0000 mgr.y (mgr.34736) 77 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:33 smithi153 bash[39117]: cluster 2024-02-10T09:44:32.859944+0000 mgr.y (mgr.34736) 77 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:34.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:33 smithi204 bash[35809]: cluster 2024-02-10T09:44:32.859944+0000 mgr.y (mgr.34736) 77 : cluster [DBG] pgmap v44: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:34] "GET /metrics HTTP/1.1" 200 153254 "" "Prometheus/2.33.4" 2024-02-10T09:44:36.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:35 smithi204 bash[35809]: cluster 2024-02-10T09:44:34.860614+0000 mgr.y (mgr.34736) 78 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:36.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:35 smithi153 bash[35780]: cluster 2024-02-10T09:44:34.860614+0000 mgr.y (mgr.34736) 78 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:36.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:35 smithi153 bash[39117]: cluster 2024-02-10T09:44:34.860614+0000 mgr.y (mgr.34736) 78 : cluster [DBG] pgmap v45: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:38.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:37 smithi204 bash[35809]: cluster 2024-02-10T09:44:36.861208+0000 mgr.y (mgr.34736) 79 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:38.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:37 smithi153 bash[35780]: cluster 2024-02-10T09:44:36.861208+0000 mgr.y (mgr.34736) 79 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:38.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:37 smithi153 bash[39117]: cluster 2024-02-10T09:44:36.861208+0000 mgr.y (mgr.34736) 79 : cluster [DBG] pgmap v46: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:40.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:39 smithi153 bash[39117]: cluster 2024-02-10T09:44:38.861583+0000 mgr.y (mgr.34736) 80 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:40.318 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:39 smithi153 bash[35780]: cluster 2024-02-10T09:44:38.861583+0000 mgr.y (mgr.34736) 80 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:40.318 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:39 smithi153 bash[66122]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:44:40.318 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:40 smithi153 bash[66122]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:44:40.319 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:40 smithi153 bash[66122]: Error: 3 errors occurred while pulling: 2024-02-10T09:44:40.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:39 smithi204 bash[35809]: cluster 2024-02-10T09:44:38.861583+0000 mgr.y (mgr.34736) 80 : cluster [DBG] pgmap v47: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:40.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:40 smithi153 bash[66122]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:44:40.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:40 smithi153 bash[66122]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:44:40.689 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:44:40 smithi153 bash[66122]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:44:41.574 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[54417]: Trying to pull registry.access.redhat.com/prom/node-exporter:latest... 2024-02-10T09:44:41.574 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[54417]: Trying to pull registry.redhat.io/prom/node-exporter:latest... 2024-02-10T09:44:41.574 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[54417]: Error: 3 errors occurred while pulling: 2024-02-10T09:44:41.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[54417]: * initializing source docker://prom/node-exporter:latest: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/node-exporter:latest: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/node-exporter:latest: reading manifest latest in docker.io/prom/node-exporter: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:44:41.831 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[54417]: * initializing source docker://registry.access.redhat.com/prom/node-exporter:latest: reading manifest latest in registry.access.redhat.com/prom/node-exporter: name unknown: Repo not found 2024-02-10T09:44:41.832 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[54417]: * initializing source docker://registry.redhat.io/prom/node-exporter:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2024-02-10T09:44:42.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:41 smithi204 bash[35809]: cluster 2024-02-10T09:44:40.862319+0000 mgr.y (mgr.34736) 81 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:42.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:41 smithi153 bash[35780]: cluster 2024-02-10T09:44:40.862319+0000 mgr.y (mgr.34736) 81 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:42.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:41 smithi153 bash[39117]: cluster 2024-02-10T09:44:40.862319+0000 mgr.y (mgr.34736) 81 : cluster [DBG] pgmap v48: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:43.957 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:44:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:44:44.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:43 smithi204 bash[35809]: cluster 2024-02-10T09:44:42.862837+0000 mgr.y (mgr.34736) 82 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:44.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:43 smithi153 bash[39117]: cluster 2024-02-10T09:44:42.862837+0000 mgr.y (mgr.34736) 82 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:44.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:43 smithi153 bash[35780]: cluster 2024-02-10T09:44:42.862837+0000 mgr.y (mgr.34736) 82 : cluster [DBG] pgmap v49: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:44.662 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:44] "GET /metrics HTTP/1.1" 200 153254 "" "Prometheus/2.33.4" 2024-02-10T09:44:45.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-10T09:44:45.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:44 smithi204 bash[35809]: audit 2024-02-10T09:44:44.385500+0000 mgr.y (mgr.34736) 83 : audit [DBG] from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:45.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:44 smithi204 bash[35809]: cluster 2024-02-10T09:44:44.863676+0000 mgr.y (mgr.34736) 84 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:45.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:44 smithi153 bash[35780]: audit 2024-02-10T09:44:44.385500+0000 mgr.y (mgr.34736) 83 : audit [DBG] from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:45.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:44 smithi153 bash[35780]: cluster 2024-02-10T09:44:44.863676+0000 mgr.y (mgr.34736) 84 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:45.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:44 smithi153 bash[39117]: audit 2024-02-10T09:44:44.385500+0000 mgr.y (mgr.34736) 83 : audit [DBG] from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:45.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:44 smithi153 bash[39117]: cluster 2024-02-10T09:44:44.863676+0000 mgr.y (mgr.34736) 84 : cluster [DBG] pgmap v50: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:47.106 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:44:47.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-10T09:44:48.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:47 smithi153 bash[35780]: cluster 2024-02-10T09:44:46.864002+0000 mgr.y (mgr.34736) 85 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:48.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:47 smithi153 bash[35780]: audit 2024-02-10T09:44:47.094580+0000 mon.a (mon.0) 979 : audit [DBG] from='client.? 172.21.15.153:0/479391042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:48.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:47 smithi153 bash[39117]: cluster 2024-02-10T09:44:46.864002+0000 mgr.y (mgr.34736) 85 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:48.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:47 smithi153 bash[39117]: audit 2024-02-10T09:44:47.094580+0000 mon.a (mon.0) 979 : audit [DBG] from='client.? 172.21.15.153:0/479391042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:47 smithi204 bash[35809]: cluster 2024-02-10T09:44:46.864002+0000 mgr.y (mgr.34736) 85 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:48.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:47 smithi204 bash[35809]: audit 2024-02-10T09:44:47.094580+0000 mon.a (mon.0) 979 : audit [DBG] from='client.? 172.21.15.153:0/479391042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:49.681 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)" 2024-02-10T09:44:50.079 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:49 smithi153 bash[35780]: cluster 2024-02-10T09:44:48.864331+0000 mgr.y (mgr.34736) 86 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:50.079 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:49 smithi153 bash[35780]: audit 2024-02-10T09:44:49.669944+0000 mon.a (mon.0) 980 : audit [DBG] from='client.? 172.21.15.153:0/3770895166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:50.080 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:49 smithi153 bash[39117]: cluster 2024-02-10T09:44:48.864331+0000 mgr.y (mgr.34736) 86 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:50.080 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:49 smithi153 bash[39117]: audit 2024-02-10T09:44:49.669944+0000 mon.a (mon.0) 980 : audit [DBG] from='client.? 172.21.15.153:0/3770895166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:49 smithi204 bash[35809]: cluster 2024-02-10T09:44:48.864331+0000 mgr.y (mgr.34736) 86 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:50.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:49 smithi204 bash[35809]: audit 2024-02-10T09:44:49.669944+0000 mon.a (mon.0) 980 : audit [DBG] from='client.? 172.21.15.153:0/3770895166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:50.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-02-10T09:44:51.976 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:51 smithi153 bash[39117]: cluster 2024-02-10T09:44:50.865002+0000 mgr.y (mgr.34736) 87 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:51.976 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:51 smithi153 bash[35780]: cluster 2024-02-10T09:44:50.865002+0000 mgr.y (mgr.34736) 87 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:52.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:51 smithi204 bash[35809]: cluster 2024-02-10T09:44:50.865002+0000 mgr.y (mgr.34736) 87 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:52.504 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:44:53.156 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:52 smithi153 bash[35780]: audit 2024-02-10T09:44:52.493134+0000 mon.a (mon.0) 981 : audit [DBG] from='client.? 172.21.15.153:0/3861972401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:53.156 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:52 smithi153 bash[39117]: audit 2024-02-10T09:44:52.493134+0000 mon.a (mon.0) 981 : audit [DBG] from='client.? 172.21.15.153:0/3861972401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:53.167 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-02-10T09:44:53.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:52 smithi204 bash[35809]: audit 2024-02-10T09:44:52.493134+0000 mon.a (mon.0) 981 : audit [DBG] from='client.? 172.21.15.153:0/3861972401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:44:53.949 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:44:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:44:54.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:53 smithi204 bash[35809]: cluster 2024-02-10T09:44:52.865350+0000 mgr.y (mgr.34736) 88 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:54.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:53 smithi153 bash[35780]: cluster 2024-02-10T09:44:52.865350+0000 mgr.y (mgr.34736) 88 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:54.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:53 smithi153 bash[39117]: cluster 2024-02-10T09:44:52.865350+0000 mgr.y (mgr.34736) 88 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:54.680 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:44:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:44:54] "GET /metrics HTTP/1.1" 200 153252 "" "Prometheus/2.33.4" 2024-02-10T09:44:56.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:55 smithi204 bash[35809]: cluster 2024-02-10T09:44:54.865961+0000 mgr.y (mgr.34736) 89 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:56.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:55 smithi153 bash[35780]: cluster 2024-02-10T09:44:54.865961+0000 mgr.y (mgr.34736) 89 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:56.423 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:55 smithi153 bash[39117]: cluster 2024-02-10T09:44:54.865961+0000 mgr.y (mgr.34736) 89 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:56.989 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:44:57.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:56 smithi204 bash[35809]: audit 2024-02-10T09:44:55.114725+0000 mgr.y (mgr.34736) 90 : audit [DBG] from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:57.388 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:56 smithi153 bash[39117]: audit 2024-02-10T09:44:55.114725+0000 mgr.y (mgr.34736) 90 : audit [DBG] from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:57.388 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:56 smithi153 bash[35780]: audit 2024-02-10T09:44:55.114725+0000 mgr.y (mgr.34736) 90 : audit [DBG] from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:44:57.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-02-10T09:44:58.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:57 smithi153 bash[39117]: cluster 2024-02-10T09:44:56.866372+0000 mgr.y (mgr.34736) 91 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:58.066 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:57 smithi153 bash[35780]: cluster 2024-02-10T09:44:56.866372+0000 mgr.y (mgr.34736) 91 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:58.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:57 smithi204 bash[35809]: cluster 2024-02-10T09:44:56.866372+0000 mgr.y (mgr.34736) 91 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:59.244 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:58 smithi153 bash[35780]: cluster 2024-02-10T09:44:58.866714+0000 mgr.y (mgr.34736) 92 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:59.244 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:58 smithi153 bash[39117]: cluster 2024-02-10T09:44:58.866714+0000 mgr.y (mgr.34736) 92 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:44:59.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:58 smithi204 bash[35809]: cluster 2024-02-10T09:44:58.866714+0000 mgr.y (mgr.34736) 92 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.610107+0000 mgr.y (mgr.34736) 93 : audit [DBG] from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:00.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.611877+0000 mon.c (mon.1) 325 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.612260+0000 mon.c (mon.1) 326 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.612541+0000 mon.c (mon.1) 327 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.612828+0000 mon.c (mon.1) 328 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.613106+0000 mon.c (mon.1) 329 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.613373+0000 mon.c (mon.1) 330 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.614931+0000 mon.c (mon.1) 331 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.615220+0000 mon.c (mon.1) 332 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.615496+0000 mon.c (mon.1) 333 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.615770+0000 mon.c (mon.1) 334 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.616051+0000 mon.c (mon.1) 335 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:44:59 smithi204 bash[35809]: audit 2024-02-10T09:44:59.933344+0000 mgr.y (mgr.34736) 94 : audit [DBG] from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mon", "hosts": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.610107+0000 mgr.y (mgr.34736) 93 : audit [DBG] from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.611877+0000 mon.c (mon.1) 325 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.612260+0000 mon.c (mon.1) 326 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.612541+0000 mon.c (mon.1) 327 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.612828+0000 mon.c (mon.1) 328 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.613106+0000 mon.c (mon.1) 329 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.613373+0000 mon.c (mon.1) 330 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.614931+0000 mon.c (mon.1) 331 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.615220+0000 mon.c (mon.1) 332 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.615496+0000 mon.c (mon.1) 333 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.615770+0000 mon.c (mon.1) 334 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.616051+0000 mon.c (mon.1) 335 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[35780]: audit 2024-02-10T09:44:59.933344+0000 mgr.y (mgr.34736) 94 : audit [DBG] from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mon", "hosts": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.610107+0000 mgr.y (mgr.34736) 93 : audit [DBG] from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.611877+0000 mon.c (mon.1) 325 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.612260+0000 mon.c (mon.1) 326 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.612541+0000 mon.c (mon.1) 327 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.612828+0000 mon.c (mon.1) 328 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.613106+0000 mon.c (mon.1) 329 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.613373+0000 mon.c (mon.1) 330 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.614931+0000 mon.c (mon.1) 331 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.615220+0000 mon.c (mon.1) 332 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.615496+0000 mon.c (mon.1) 333 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.615770+0000 mon.c (mon.1) 334 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.616051+0000 mon.c (mon.1) 335 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:00.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:44:59 smithi153 bash[39117]: audit 2024-02-10T09:44:59.933344+0000 mgr.y (mgr.34736) 94 : audit [DBG] from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mon", "hosts": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:01.143 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:00 smithi153 bash[35780]: cluster 2024-02-10T09:45:00.867287+0000 mgr.y (mgr.34736) 95 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:01.143 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:00 smithi153 bash[39117]: cluster 2024-02-10T09:45:00.867287+0000 mgr.y (mgr.34736) 95 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:01.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:00 smithi204 bash[35809]: cluster 2024-02-10T09:45:00.867287+0000 mgr.y (mgr.34736) 95 : cluster [DBG] pgmap v58: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:01.575 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: cephadm 2024-02-10T09:45:01.571043+0000 mgr.y (mgr.34736) 96 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.571652+0000 mon.c (mon.1) 336 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.571980+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.574932+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.625709+0000 mon.c (mon.1) 337 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.919272+0000 mon.c (mon.1) 338 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.919564+0000 mon.a (mon.0) 984 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: audit 2024-02-10T09:45:01.923018+0000 mon.a (mon.0) 985 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[35780]: cephadm 2024-02-10T09:45:01.928632+0000 mgr.y (mgr.34736) 97 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: cephadm 2024-02-10T09:45:01.571043+0000 mgr.y (mgr.34736) 96 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.571652+0000 mon.c (mon.1) 336 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.571980+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.574932+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.625709+0000 mon.c (mon.1) 337 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.919272+0000 mon.c (mon.1) 338 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.919564+0000 mon.a (mon.0) 984 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: audit 2024-02-10T09:45:01.923018+0000 mon.a (mon.0) 985 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:01 smithi153 bash[39117]: cephadm 2024-02-10T09:45:01.928632+0000 mgr.y (mgr.34736) 97 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:02.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:45:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: cephadm 2024-02-10T09:45:01.571043+0000 mgr.y (mgr.34736) 96 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.571652+0000 mon.c (mon.1) 336 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.571980+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.574932+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.625709+0000 mon.c (mon.1) 337 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:02.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.919272+0000 mon.c (mon.1) 338 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.919564+0000 mon.a (mon.0) 984 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: audit 2024-02-10T09:45:01.923018+0000 mon.a (mon.0) 985 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:02.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:01 smithi204 bash[35809]: cephadm 2024-02-10T09:45:01.928632+0000 mgr.y (mgr.34736) 97 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:03.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:02 smithi204 bash[35809]: cluster 2024-02-10T09:45:02.867631+0000 mgr.y (mgr.34736) 98 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:03.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:02 smithi153 bash[39117]: cluster 2024-02-10T09:45:02.867631+0000 mgr.y (mgr.34736) 98 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:03.418 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:02 smithi153 bash[35780]: cluster 2024-02-10T09:45:02.867631+0000 mgr.y (mgr.34736) 98 : cluster [DBG] pgmap v59: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:45:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:04] "GET /metrics HTTP/1.1" 200 153251 "" "Prometheus/2.33.4" 2024-02-10T09:45:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.843338+0000 mon.c (mon.1) 339 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.843699+0000 mon.a (mon.0) 986 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.847088+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.848379+0000 mon.c (mon.1) 340 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.849192+0000 mon.c (mon.1) 341 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:05.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.849409+0000 mon.a (mon.0) 988 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.853322+0000 mon.a (mon.0) 989 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.854633+0000 mon.c (mon.1) 342 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.843338+0000 mon.c (mon.1) 339 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.843699+0000 mon.a (mon.0) 986 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.847088+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.848379+0000 mon.c (mon.1) 340 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.849192+0000 mon.c (mon.1) 341 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.849409+0000 mon.a (mon.0) 988 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.853322+0000 mon.a (mon.0) 989 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:45:05.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.854633+0000 mon.c (mon.1) 342 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.855376+0000 mon.c (mon.1) 343 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.855597+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.859139+0000 mon.a (mon.0) 991 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.860106+0000 mon.c (mon.1) 344 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[35780]: audit 2024-02-10T09:45:04.860687+0000 mon.c (mon.1) 345 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.855376+0000 mon.c (mon.1) 343 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.855597+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.859139+0000 mon.a (mon.0) 991 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:45:05.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.860106+0000 mon.c (mon.1) 344 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:45:05.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:04 smithi153 bash[39117]: audit 2024-02-10T09:45:04.860687+0000 mon.c (mon.1) 345 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T09:45:05.279 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:45:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.843338+0000 mon.c (mon.1) 339 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:05.331 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.843699+0000 mon.a (mon.0) 986 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.847088+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi204\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.848379+0000 mon.c (mon.1) 340 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.849192+0000 mon.c (mon.1) 341 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.849409+0000 mon.a (mon.0) 988 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.853322+0000 mon.a (mon.0) 989 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.854633+0000 mon.c (mon.1) 342 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.855376+0000 mon.c (mon.1) 343 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.855597+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.859139+0000 mon.a (mon.0) 991 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.860106+0000 mon.c (mon.1) 344 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:45:05.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:04 smithi204 bash[35809]: audit 2024-02-10T09:45:04.860687+0000 mon.c (mon.1) 345 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (13m) 62s ago 13m 26.0M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (54s) 49s ago 16m 25.7M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (84s) 49s ago 26m 385M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (9m) 62s ago 28m 427M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (28m) 62s ago 28m 204M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (26m) 49s ago 26m 189M 2048M 15.2.0 204a01f9b0b6 c3b5b5ecd32c 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (27m) 62s ago 27m 185M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (72s) 62s ago 18m 11.9M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:45:05.887 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (67s) 49s ago 18m 12.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (25m) 62s ago 26m 43.8M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (25m) 62s ago 25m 44.4M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (25m) 62s ago 25m 43.1M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (25m) 62s ago 25m 46.0M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (24m) 49s ago 24m 47.8M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (24m) 49s ago 24m 48.7M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (24m) 49s ago 24m 46.5M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (23m) 49s ago 24m 45.5M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (13m) 49s ago 13m 66.0M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (16m) 62s ago 16m 35.9M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:45:05.888 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (16m) 49s ago 16m 37.1M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: cephadm 2024-02-10T09:45:04.847479+0000 mgr.y (mgr.34736) 99 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: cephadm 2024-02-10T09:45:04.847537+0000 mgr.y (mgr.34736) 100 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: cephadm 2024-02-10T09:45:04.855209+0000 mgr.y (mgr.34736) 101 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: cephadm 2024-02-10T09:45:04.861098+0000 mgr.y (mgr.34736) 102 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: cluster 2024-02-10T09:45:04.868225+0000 mgr.y (mgr.34736) 103 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.877240+0000 mon.c (mon.1) 346 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.877580+0000 mon.c (mon.1) 347 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:05.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.877856+0000 mon.c (mon.1) 348 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.878127+0000 mon.c (mon.1) 349 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.878387+0000 mon.c (mon.1) 350 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.878642+0000 mon.c (mon.1) 351 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.880245+0000 mon.c (mon.1) 352 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.880533+0000 mon.c (mon.1) 353 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.880808+0000 mon.c (mon.1) 354 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.881082+0000 mon.c (mon.1) 355 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:05.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:05 smithi204 bash[35809]: audit 2024-02-10T09:45:05.881333+0000 mon.c (mon.1) 356 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: cephadm 2024-02-10T09:45:04.847479+0000 mgr.y (mgr.34736) 99 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: cephadm 2024-02-10T09:45:04.847537+0000 mgr.y (mgr.34736) 100 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: cephadm 2024-02-10T09:45:04.855209+0000 mgr.y (mgr.34736) 101 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: cephadm 2024-02-10T09:45:04.861098+0000 mgr.y (mgr.34736) 102 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: cluster 2024-02-10T09:45:04.868225+0000 mgr.y (mgr.34736) 103 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.877240+0000 mon.c (mon.1) 346 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.877580+0000 mon.c (mon.1) 347 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.877856+0000 mon.c (mon.1) 348 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.878127+0000 mon.c (mon.1) 349 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.878387+0000 mon.c (mon.1) 350 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.878642+0000 mon.c (mon.1) 351 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.880245+0000 mon.c (mon.1) 352 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.880533+0000 mon.c (mon.1) 353 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.880808+0000 mon.c (mon.1) 354 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.881082+0000 mon.c (mon.1) 355 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[35780]: audit 2024-02-10T09:45:05.881333+0000 mon.c (mon.1) 356 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: cephadm 2024-02-10T09:45:04.847479+0000 mgr.y (mgr.34736) 99 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: cephadm 2024-02-10T09:45:04.847537+0000 mgr.y (mgr.34736) 100 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: cephadm 2024-02-10T09:45:04.855209+0000 mgr.y (mgr.34736) 101 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: cephadm 2024-02-10T09:45:04.861098+0000 mgr.y (mgr.34736) 102 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2024-02-10T09:45:06.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: cluster 2024-02-10T09:45:04.868225+0000 mgr.y (mgr.34736) 103 : cluster [DBG] pgmap v60: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.877240+0000 mon.c (mon.1) 346 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.877580+0000 mon.c (mon.1) 347 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.877856+0000 mon.c (mon.1) 348 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.878127+0000 mon.c (mon.1) 349 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.878387+0000 mon.c (mon.1) 350 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.878642+0000 mon.c (mon.1) 351 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.880245+0000 mon.c (mon.1) 352 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.880533+0000 mon.c (mon.1) 353 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.880808+0000 mon.c (mon.1) 354 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.881082+0000 mon.c (mon.1) 355 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:05 smithi153 bash[39117]: audit 2024-02-10T09:45:05.881333+0000 mon.c (mon.1) 356 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:45:06.266 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 13, 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:45:06.267 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi204", 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/1 daemons upgraded", 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T09:45:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:45:06.576 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:45:07.081 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:05.267623+0000 mgr.y (mgr.34736) 104 : audit [DBG] from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:05.569802+0000 mgr.y (mgr.34736) 105 : audit [DBG] from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:05.875352+0000 mgr.y (mgr.34736) 106 : audit [DBG] from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.265814+0000 mon.c (mon.1) 357 : audit [DBG] from='client.? 172.21.15.153:0/981536443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.377332+0000 mon.c (mon.1) 358 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.377606+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.381682+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]': finished 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.382457+0000 mon.c (mon.1) 359 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.383018+0000 mon.c (mon.1) 360 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.383592+0000 mon.c (mon.1) 361 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:45:07.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:06 smithi204 bash[35809]: audit 2024-02-10T09:45:06.384645+0000 mon.c (mon.1) 362 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:45:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:05.267623+0000 mgr.y (mgr.34736) 104 : audit [DBG] from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:05.569802+0000 mgr.y (mgr.34736) 105 : audit [DBG] from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:05.875352+0000 mgr.y (mgr.34736) 106 : audit [DBG] from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.265814+0000 mon.c (mon.1) 357 : audit [DBG] from='client.? 172.21.15.153:0/981536443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.377332+0000 mon.c (mon.1) 358 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.377606+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.381682+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]': finished 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.382457+0000 mon.c (mon.1) 359 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.383018+0000 mon.c (mon.1) 360 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.383592+0000 mon.c (mon.1) 361 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[35780]: audit 2024-02-10T09:45:06.384645+0000 mon.c (mon.1) 362 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:05.267623+0000 mgr.y (mgr.34736) 104 : audit [DBG] from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:05.569802+0000 mgr.y (mgr.34736) 105 : audit [DBG] from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:05.875352+0000 mgr.y (mgr.34736) 106 : audit [DBG] from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:07.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.265814+0000 mon.c (mon.1) 357 : audit [DBG] from='client.? 172.21.15.153:0/981536443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.377332+0000 mon.c (mon.1) 358 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]: dispatch 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.377606+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]: dispatch 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.381682+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.b"}]': finished 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.382457+0000 mon.c (mon.1) 359 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.383018+0000 mon.c (mon.1) 360 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.383592+0000 mon.c (mon.1) 361 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:45:07.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:06 smithi153 bash[39117]: audit 2024-02-10T09:45:06.384645+0000 mon.c (mon.1) 362 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:45:08.123 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:07 smithi204 bash[35809]: cephadm 2024-02-10T09:45:06.376887+0000 mgr.y (mgr.34736) 107 : cephadm [INF] Upgrade: Updating mon.b 2024-02-10T09:45:08.124 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:07 smithi204 bash[35809]: cephadm 2024-02-10T09:45:06.384187+0000 mgr.y (mgr.34736) 108 : cephadm [INF] Deploying daemon mon.b on smithi204 2024-02-10T09:45:08.124 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:07 smithi204 bash[35809]: audit 2024-02-10T09:45:06.574704+0000 mgr.y (mgr.34736) 109 : audit [DBG] from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:08.124 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:07 smithi204 bash[35809]: cluster 2024-02-10T09:45:06.868719+0000 mgr.y (mgr.34736) 110 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:08.124 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:08 smithi204 systemd[1]: Stopping Ceph mon.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:45:08.124 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:08 smithi204 bash[35809]: debug 2024-02-10T09:45:08.122+0000 7f4cd7005700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:45:08.125 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:08 smithi204 bash[35809]: debug 2024-02-10T09:45:08.122+0000 7f4cd7005700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T09:45:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[35780]: cephadm 2024-02-10T09:45:06.376887+0000 mgr.y (mgr.34736) 107 : cephadm [INF] Upgrade: Updating mon.b 2024-02-10T09:45:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[35780]: cephadm 2024-02-10T09:45:06.384187+0000 mgr.y (mgr.34736) 108 : cephadm [INF] Deploying daemon mon.b on smithi204 2024-02-10T09:45:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[35780]: audit 2024-02-10T09:45:06.574704+0000 mgr.y (mgr.34736) 109 : audit [DBG] from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:08.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[35780]: cluster 2024-02-10T09:45:06.868719+0000 mgr.y (mgr.34736) 110 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:08.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[39117]: cephadm 2024-02-10T09:45:06.376887+0000 mgr.y (mgr.34736) 107 : cephadm [INF] Upgrade: Updating mon.b 2024-02-10T09:45:08.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[39117]: cephadm 2024-02-10T09:45:06.384187+0000 mgr.y (mgr.34736) 108 : cephadm [INF] Deploying daemon mon.b on smithi204 2024-02-10T09:45:08.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[39117]: audit 2024-02-10T09:45:06.574704+0000 mgr.y (mgr.34736) 109 : audit [DBG] from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:08.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:07 smithi153 bash[39117]: cluster 2024-02-10T09:45:06.868719+0000 mgr.y (mgr.34736) 110 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:08.831 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:08 smithi204 bash[59903]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.b 2024-02-10T09:45:09.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.b.service: Succeeded. 2024-02-10T09:45:09.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 systemd[1]: Stopped Ceph mon.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:45:09.832 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 systemd[1]: Starting Ceph mon.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:45:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.965+0000 7fa02991b880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:45:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.965+0000 7fa02991b880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-10T09:45:10.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.965+0000 7fa02991b880 0 pidfile_write: ignore empty --pid-file 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.970+0000 7fa02991b880 0 load: jerasure load: lrc load: isa 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: DB SUMMARY 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: CURRENT file: CURRENT 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 2825 Bytes 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 3, files: 000064.sst 000065.sst 000066.sst 2024-02-10T09:45:10.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000062.log size: 7703342 ; 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.create_if_missing: 0 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.env: 0x559fcd88b080 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.fs: Posix File System 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.info_log: 0x559fcea70260 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:45:10.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.db_log_dir: 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:45:10.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.write_buffer_manager: 0x559fcf762390 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:45:10.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.unordered_write: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.row_cache: None 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.wal_filter: None 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:45:10.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:45:10.338 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Compression algorithms supported: 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kZSTD supported: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:45:10.339 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.merge_operator: 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559fcea15d28) 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cache_index_and_filter_blocks: 1 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:45:10.340 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: pin_top_level_index_and_filter: 1 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: index_type: 0 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: data_block_index_type: 0 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: index_shortening: 1 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: hash_index_allow_collision: 1 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: checksum: 1 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: no_block_cache: 0 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_cache: 0x559fcea4af10 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_cache_name: BinnedLRUCache 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_cache_options: 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: capacity : 536870912 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: num_shard_bits : 4 2024-02-10T09:45:10.341 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: strict_capacity_limit : 0 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: high_pri_pool_ratio: 0.000 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_cache_compressed: (nil) 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: persistent_cache: (nil) 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_size: 4096 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_size_deviation: 10 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_restart_interval: 16 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: index_block_restart_interval: 1 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: metadata_block_size: 4096 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: partition_filters: 0 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: use_delta_encoding: 1 2024-02-10T09:45:10.342 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:45:10.343 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: whole_key_filtering: 1 2024-02-10T09:45:10.343 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: verify_compression: 0 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: read_amp_bytes_per_bit: 0 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: format_version: 2 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: enable_index_compression: 1 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: block_align: 0 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.num_levels: 7 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:45:10.344 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:45:10.345 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:45:10.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:45:10.347 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:45:10.348 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.ttl: 2592000 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.971+0000 7fa02991b880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.972+0000 7fa02991b880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 68, last_sequence is 24306, log_number is 62,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.972+0000 7fa02991b880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 62 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.972+0000 7fa02991b880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558309972439, "job": 1, "event": "recovery_started", "log_files": [62]} 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.972+0000 7fa02991b880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #62 mode 2 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.978+0000 7fa02991b880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.985+0000 7fa02991b880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558309985554, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 68, "file_size": 5128947, "table_properties": {"data_size": 5123924, "index_size": 3624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 3838, "raw_average_key_size": 21, "raw_value_size": 5118559, "raw_average_value_size": 28279, "num_data_blocks": 112, "num_entries": 181, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707558309, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T09:45:10.349 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.985+0000 7fa02991b880 4 rocksdb: [version_set.cc:3826] Creating manifest 69 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.986+0000 7fa02991b880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558309986163, "job": 1, "event": "recovery_finished"} 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.987+0000 7fa02991b880 4 rocksdb: DB pointer 0x559fceb13800 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.987+0000 7fa01070c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.987+0000 7fa01070c700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** DB Stats ** 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.350 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** Compaction Stats [default] ** 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: L0 1/0 4.89 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: L6 3/0 129.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Sum 4/0 134.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** Compaction Stats [default] ** 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Flush(GB): cumulative 0.005, interval 0.005 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:45:10.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Cumulative compaction: 0.00 GB write, 307.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Interval compaction: 0.00 GB write, 307.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** Compaction Stats [default] ** 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: L0 1/0 4.89 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: L6 3/0 129.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:45:10.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Sum 4/0 134.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** Compaction Stats [default] ** 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 668.9 0.01 0.00 1 0.007 0 0 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Flush(GB): cumulative 0.005, interval 0.000 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Cumulative compaction: 0.00 GB write, 306.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:10.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.987+0000 7fa02991b880 0 starting mon.b rank 2 at public addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] at bind addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.988+0000 7fa02991b880 1 mon.b@-1(???) e3 preinit fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 0 mon.b@-1(???).mds e1 new map 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 0 mon.b@-1(???).mds e1 print_map 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: e1 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: enable_multiple, ever_enabled_multiple: 0,0 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: legacy client fscid: -1 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: No filesystems configured 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 2024-02-10T09:45:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 0 mon.b@-1(???).osd e72 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 0 mon.b@-1(???).osd e72 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 0 mon.b@-1(???).osd e72 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 0 mon.b@-1(???).osd e72 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cephadm 2024-02-10T09:45:06.376887+0000 mgr.y (mgr.34736) 107 : cephadm [INF] Upgrade: Updating mon.b 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cephadm 2024-02-10T09:45:06.384187+0000 mgr.y (mgr.34736) 108 : cephadm [INF] Deploying daemon mon.b on smithi204 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: audit 2024-02-10T09:45:06.574704+0000 mgr.y (mgr.34736) 109 : audit [DBG] from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:06.868719+0000 mgr.y (mgr.34736) 110 : cluster [DBG] pgmap v61: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:45:09.989+0000 7fa02991b880 1 mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:10 smithi204 bash[60088]: a2615d19aa22819ee2e250758e9fa7f544e03b2c363790ef6027ddcb9938eeb0 2024-02-10T09:45:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:10 smithi204 systemd[1]: Started Ceph mon.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:08.869054+0000 mgr.y (mgr.34736) 111 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: audit 2024-02-10T09:45:10.064067+0000 mon.c (mon.1) 363 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.196276+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.196965+0000 mon.c (mon.1) 364 : cluster [INF] mon.c calling monitor election 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: audit 2024-02-10T09:45:10.197160+0000 mon.a (mon.0) 995 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"pr 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: ometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: audit 2024-02-10T09:45:10.197575+0000 mon.a (mon.0) 996 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.197619+0000 mon.a (mon.0) 997 : cluster [INF] mon.a calling monitor election 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.211464+0000 mon.a (mon.0) 998 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: audit 2024-02-10T09:45:10.213499+0000 mon.a (mon.0) 999 : audit [INF] from='mgr.34736 ' entity='' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-1 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: 0T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.219590+0000 mon.a (mon.0) 1000 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.219667+0000 mon.a (mon.0) 1001 : cluster [DBG] fsmap 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.219718+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: cluster 2024-02-10T09:45:10.220473+0000 mon.a (mon.0) 1003 : cluster [DBG] mgrmap e39: y(active, since 118s), standbys: x 2024-02-10T09:45:11.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:45:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: audit 2024-02-10T09:45:10.221301+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devic 2024-02-10T09:45:11.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:08.869054+0000 mgr.y (mgr.34736) 111 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:11.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.064067+0000 mon.c (mon.1) 363 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.196276+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T09:45:11.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.196965+0000 mon.c (mon.1) 364 : cluster [INF] mon.c calling monitor election 2024-02-10T09:45:11.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.197160+0000 mon.a (mon.0) 995 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:08.869054+0000 mgr.y (mgr.34736) 111 : cluster [DBG] pgmap v62: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:11.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.197575+0000 mon.a (mon.0) 996 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.197619+0000 mon.a (mon.0) 997 : cluster [INF] mon.a calling monitor election 2024-02-10T09:45:11.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.211464+0000 mon.a (mon.0) 998 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:45:11.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.064067+0000 mon.c (mon.1) 363 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.196276+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T09:45:11.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.196965+0000 mon.c (mon.1) 364 : cluster [INF] mon.c calling monitor election 2024-02-10T09:45:11.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.197160+0000 mon.a (mon.0) 995 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.213499+0000 mon.a (mon.0) 999 : audit [INF] from='mgr.34736 ' entity='' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.219590+0000 mon.a (mon.0) 1000 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:45:11.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.219667+0000 mon.a (mon.0) 1001 : cluster [DBG] fsmap 2024-02-10T09:45:11.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.219718+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:45:11.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.220473+0000 mon.a (mon.0) 1003 : cluster [DBG] mgrmap e39: y(active, since 118s), standbys: x 2024-02-10T09:45:11.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.197575+0000 mon.a (mon.0) 996 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.197619+0000 mon.a (mon.0) 997 : cluster [INF] mon.a calling monitor election 2024-02-10T09:45:11.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.211464+0000 mon.a (mon.0) 998 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.221301+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.225518+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.34736 ' entity='' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.225672+0000 mon.a (mon.0) 1006 : cluster [INF] overall HEALTH_OK 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.226897+0000 mon.c (mon.1) 365 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.229411+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: audit 2024-02-10T09:45:10.231423+0000 mon.c (mon.1) 366 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:45:11.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[35780]: cluster 2024-02-10T09:45:10.870000+0000 mgr.y (mgr.34736) 112 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:11.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.213499+0000 mon.a (mon.0) 999 : audit [INF] from='mgr.34736 ' entity='' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.219590+0000 mon.a (mon.0) 1000 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} 2024-02-10T09:45:11.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.219667+0000 mon.a (mon.0) 1001 : cluster [DBG] fsmap 2024-02-10T09:45:11.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.219718+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.220473+0000 mon.a (mon.0) 1003 : cluster [DBG] mgrmap e39: y(active, since 118s), standbys: x 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.221301+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.225518+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.34736 ' entity='' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.225672+0000 mon.a (mon.0) 1006 : cluster [INF] overall HEALTH_OK 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.226897+0000 mon.c (mon.1) 365 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.229411+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 404121190, \"cpu_percentage\": \"21.68%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:16.846463Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 50142904, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846552Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 51170508, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846633Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48811212, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846711Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 47804579, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.846841Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12635340, \"cpu_percentage\": \"0.85%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.846925Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 27053260, \"cpu_percentage\": \"55.42%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:16.847007Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 38986055, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:16.847102Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\", \"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\"], \"memory_usage\": 69216501, \"cpu_percentage\": \"0.29%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:44:16.847180Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: audit 2024-02-10T09:45:10.231423+0000 mon.c (mon.1) 366 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:45:11.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:11 smithi153 bash[39117]: cluster 2024-02-10T09:45:10.870000+0000 mgr.y (mgr.34736) 112 : cluster [DBG] pgmap v63: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:45:14.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:13 smithi153 bash[35780]: cluster 2024-02-10T09:45:12.870375+0000 mgr.y (mgr.34736) 113 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:14.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:13 smithi153 bash[39117]: cluster 2024-02-10T09:45:12.870375+0000 mgr.y (mgr.34736) 113 : cluster [DBG] pgmap v64: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:14] "GET /metrics HTTP/1.1" 200 153251 "" "Prometheus/2.33.4" 2024-02-10T09:45:15.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:14.994+0000 7f12bff08700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-10T09:45:16.341 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: cluster 2024-02-10T09:45:14.871243+0000 mgr.y (mgr.34736) 114 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:16.342 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: cluster 2024-02-10T09:45:14.871243+0000 mgr.y (mgr.34736) 114 : cluster [DBG] pgmap v65: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:16.342 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.001591+0000 mon.c (mon.1) 367 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"a2615d19aa22\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 25490882, \"cpu_percentage\": \"13.60%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999222Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:45:09.607793Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 405589196, \"cpu_percentage\": \"8.15%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:45:14.999403Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50205818, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999493Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51201966, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999576Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49199185, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999656Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47836037, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999770Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 14900264, \"cpu_percentage\": \"0.71%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999864Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 51600424, \"cpu_percentage\": \"1.45%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999946Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 39478886, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:15.000027Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69237473, \"cpu_percentage\": \"0.30%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:45:15.000120Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:45:15.000278Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:16.343 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.001591+0000 mon.c (mon.1) 367 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"a2615d19aa22\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 25490882, \"cpu_percentage\": \"13.60%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999222Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:45:09.607793Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 405589196, \"cpu_percentage\": \"8.15%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:45:14.999403Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50205818, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999493Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51201966, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999576Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49199185, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999656Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47836037, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999770Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 14900264, \"cpu_percentage\": \"0.71%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999864Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 51600424, \"cpu_percentage\": \"1.45%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999946Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 39478886, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:15.000027Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69237473, \"cpu_percentage\": \"0.30%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:45:15.000120Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:45:15.000278Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:16.343 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.002226+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"a2615d19aa22\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 25490882, \"cpu_percentage\": \"13.60%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999222Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:45:09.607793Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 405589196, \"cpu_percentage\": \"8.15%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:45:14.999403Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50205818, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999493Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51201966, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999576Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49199185, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999656Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47836037, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999770Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 14900264, \"cpu_percentage\": \"0.71%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999864Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 51600424, \"cpu_percentage\": \"1.45%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999946Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 39478886, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:15.000027Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69237473, \"cpu_percentage\": \"0.30%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:45:15.000120Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:45:15.000278Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:16.343 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.002226+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"a2615d19aa22\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 25490882, \"cpu_percentage\": \"13.60%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999222Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:45:09.607793Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 405589196, \"cpu_percentage\": \"8.15%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:45:14.999403Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50205818, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999493Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51201966, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999576Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49199185, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999656Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47836037, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999770Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 14900264, \"cpu_percentage\": \"0.71%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999864Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 51600424, \"cpu_percentage\": \"1.45%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999946Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 39478886, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:15.000027Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69237473, \"cpu_percentage\": \"0.30%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:45:15.000120Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:45:15.000278Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:16.343 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.009165+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"a2615d19aa22\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 25490882, \"cpu_percentage\": \"13.60%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999222Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:45:09.607793Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 405589196, \"cpu_percentage\": \"8.15%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:45:14.999403Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50205818, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999493Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51201966, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999576Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49199185, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999656Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47836037, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999770Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 14900264, \"cpu_percentage\": \"0.71%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999864Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 51600424, \"cpu_percentage\": \"1.45%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999946Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 39478886, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:15.000027Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69237473, \"cpu_percentage\": \"0.30%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:45:15.000120Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:45:15.000278Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.011336+0000 mon.c (mon.1) 368 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.011677+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.017348+0000 mon.a (mon.0) 1011 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.024065+0000 mon.c (mon.1) 369 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.025123+0000 mon.c (mon.1) 370 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.025404+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.026778+0000 mon.c (mon.1) 371 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.009165+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi204","val":"{\"daemons\": {\"mon.b\": {\"daemon_type\": \"mon\", \"daemon_id\": \"b\", \"service_name\": \"mon\", \"daemon_name\": \"mon.b\", \"hostname\": \"smithi204\", \"container_id\": \"a2615d19aa22\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 25490882, \"cpu_percentage\": \"13.60%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999222Z\", \"created\": \"2024-02-10T09:18:21.548104Z\", \"started\": \"2024-02-10T09:45:09.607793Z\"}, \"mgr.x\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"x\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.x\", \"hostname\": \"smithi204\", \"container_id\": \"c26e3d68d449\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 405589196, \"cpu_percentage\": \"8.15%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:45:14.999403Z\", \"created\": \"2024-02-10T09:18:35.961903Z\", \"started\": \"2024-02-10T09:43:40.901004Z\"}, \"osd.4\": {\"daemon_type\": \"osd\", \"daemon_id\": \"4\", \"daemon_name\": \"osd.4\", \"hostname\": \"smithi204\", \"container_id\": \"d707da0e0933\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 50205818, \"cpu_percentage\": \"0.55%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999493Z\", \"created\": \"2024-02-10T09:20:10.443566Z\", \"started\": \"2024-02-10T09:20:12.035540Z\"}, \"osd.5\": {\"daemon_type\": \"osd\", \"daemon_id\": \"5\", \"daemon_name\": \"osd.5\", \"hostname\": \"smithi204\", \"container_id\": \"5d6ec7ebaa7b\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 51201966, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999576Z\", \"created\": \"2024-02-10T09:20:28.489307Z\", \"started\": \"2024-02-10T09:20:30.785766Z\"}, \"osd.6\": {\"daemon_type\": \"osd\", \"daemon_id\": \"6\", \"daemon_name\": \"osd.6\", \"hostname\": \"smithi204\", \"container_id\": \"cb0fc1977562\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 49199185, \"cpu_percentage\": \"0.47%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999656Z\", \"created\": \"2024-02-10T09:20:47.002041Z\", \"started\": \"2024-02-10T09:20:49.479446Z\"}, \"osd.7\": {\"daemon_type\": \"osd\", \"daemon_id\": \"7\", \"daemon_name\": \"osd.7\", \"hostname\": \"smithi204\", \"container_id\": \"553a3da04795\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 47836037, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:14.999770Z\", \"created\": \"2024-02-10T09:21:05.721771Z\", \"started\": \"2024-02-10T09:21:07.713166Z\"}, \"node-exporter.b\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"b\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.b\", \"hostname\": \"smithi204\", \"container_id\": \"accc637b7e99\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 14900264, \"cpu_percentage\": \"0.71%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999864Z\", \"created\": \"2024-02-10T09:26:14.659269Z\", \"started\": \"2024-02-10T09:43:58.231240Z\"}, \"grafana.a\": {\"daemon_type\": \"grafana\", \"daemon_id\": \"a\", \"service_name\": \"grafana\", \"daemon_name\": \"grafana.a\", \"hostname\": \"smithi204\", \"container_id\": \"67a3139af0c0\", \"container_image_id\": \"dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01\", \"container_image_name\": \"quay.io/ceph/ceph-grafana:8.3.5\", \"container_image_digests\": [\"quay.io/ceph/ceph-grafana@sha256:f5273853535cbeb49c24ab539ad6e21ca41c9e894753f33b992f232ce40d6c00\", \"quay.io/ceph/ceph-grafana@sha256:09d5d36bc52314dc0f86e616d959d147bb61d4120dcce1f9be244b324c4b0f36\"], \"memory_usage\": 51600424, \"cpu_percentage\": \"1.45%\", \"version\": \"8.3.5\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:45:14.999946Z\", \"created\": \"2024-02-10T09:29:00.434804Z\", \"started\": \"2024-02-10T09:44:11.347753Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi204.kmhqqg\", \"daemon_name\": \"rgw.r.z.smithi204.kmhqqg\", \"hostname\": \"smithi204\", \"container_id\": \"a9860a65eacb\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\", \"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\"], \"memory_usage\": 39478886, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:45:15.000027Z\", \"created\": \"2024-02-10T09:29:04.291747Z\", \"started\": \"2024-02-10T09:29:04.321392Z\"}, \"prometheus.a\": {\"daemon_type\": \"prometheus\", \"daemon_id\": \"a\", \"service_name\": \"prometheus\", \"daemon_name\": \"prometheus.a\", \"hostname\": \"smithi204\", \"container_id\": \"ac1f43abcfcc\", \"container_image_id\": \"514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d\", \"container_image_name\": \"quay.io/prometheus/prometheus:v2.33.4\", \"container_image_digests\": [\"quay.io/prometheus/prometheus@sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c\", \"quay.io/prometheus/prometheus@sha256:35337321e0e7d8c62b5035b712f895170fe2ec42d3181f0d46a7d1af3dfdb1d4\"], \"memory_usage\": 69237473, \"cpu_percentage\": \"0.30%\", \"version\": \"2.33.4\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [9095], \"last_refresh\": \"2024-02-10T09:45:15.000120Z\", \"created\": \"2024-02-10T09:31:48.243379Z\", \"started\": \"2024-02-10T09:31:47.711075Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946585Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"7\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"d38f33ce-8c0b-4235-be4f-4169e07e4cde\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"VEZq29-kij4-KLsl-tGBd-zvEt-wl0d-UCT1gv\"}, {\"name\": \"lv_2\", \"osd_id\": \"6\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"2f3e1b43-d7dd-44b2-be39-d240e518a51c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"cCX1ed-Oc2D-4Y7A-3GcS-zziZ-UwoD-dPZxN7\"}, {\"name\": \"lv_3\", \"osd_id\": \"5\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"3ed1e885-ba58-4d66-a134-e63d325d4fcb\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"nuYS0e-1NAE-UUTr-LK8c-N45a-Reu2-sEDuFw\"}, {\"name\": \"lv_4\", \"osd_id\": \"4\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"4e6b26a2-2084-4ce3-a544-5f4e186ff1a0\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"flkeGT-qTnp-FUkz-oLeR-L3jT-DErl-LgdARX\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300GD400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.946594Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XP2E\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:06.384054Z\"}, \"mgr.x\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:27.943526Z\"}, \"osd.4\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:08.905357Z\"}, \"osd.5\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:26.993402Z\"}, \"osd.6\": {\"deps\": [], \"last_config\": \"2024-02-10T09:20:45.393328Z\"}, \"osd.7\": {\"deps\": [], \"last_config\": \"2024-02-10T09:21:04.063785Z\"}, \"node-exporter.b\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:55.084078Z\"}, \"grafana.a\": {\"deps\": [\"prometheus.a\"], \"last_config\": \"2024-02-10T09:44:07.829602Z\"}, \"rgw.r.z.smithi204.kmhqqg\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:02.675775Z\"}, \"prometheus.a\": {\"deps\": [\"9283\", \"alertmanager.a\", \"mgr.x\", \"mgr.y\", \"node-exporter.a\", \"node-exporter.b\"], \"last_config\": \"2024-02-10T09:31:26.354152Z\"}}, \"last_daemon_update\": \"2024-02-10T09:45:15.000278Z\", \"last_device_update\": \"2024-02-10T09:21:11.911631Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.483677Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]': finished 2024-02-10T09:45:16.344 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.011336+0000 mon.c (mon.1) 368 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.011677+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.017348+0000 mon.a (mon.0) 1011 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.024065+0000 mon.c (mon.1) 369 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.025123+0000 mon.c (mon.1) 370 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.025404+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.026778+0000 mon.c (mon.1) 371 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.028248+0000 mon.c (mon.1) 372 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.345 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.029077+0000 mon.c (mon.1) 373 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.029311+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.035783+0000 mon.a (mon.0) 1014 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]': finished 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.053767+0000 mon.c (mon.1) 374 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.054627+0000 mon.c (mon.1) 375 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.054877+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.065975+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]': finished 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.068687+0000 mon.c (mon.1) 376 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.070090+0000 mon.c (mon.1) 377 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.070470+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.078766+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.080464+0000 mon.c (mon.1) 378 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.081773+0000 mon.c (mon.1) 379 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.082129+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.087633+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]': finished 2024-02-10T09:45:16.346 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.088838+0000 mon.c (mon.1) 380 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.089157+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.214891+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.241579+0000 mon.c (mon.1) 381 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.242491+0000 mon.c (mon.1) 382 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.242829+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.247926+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]': finished 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.249417+0000 mon.c (mon.1) 383 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.250506+0000 mon.c (mon.1) 384 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.250822+0000 mon.a (mon.0) 1025 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.028248+0000 mon.c (mon.1) 372 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.029077+0000 mon.c (mon.1) 373 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.347 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.029311+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.035783+0000 mon.a (mon.0) 1014 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.crash"}]': finished 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.053767+0000 mon.c (mon.1) 374 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.054627+0000 mon.c (mon.1) 375 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.054877+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.065975+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mds"}]': finished 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.068687+0000 mon.c (mon.1) 376 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.070090+0000 mon.c (mon.1) 377 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.070470+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.078766+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:45:16.348 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.080464+0000 mon.c (mon.1) 378 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.255908+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]': finished 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.257717+0000 mon.c (mon.1) 385 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.258868+0000 mon.c (mon.1) 386 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.259195+0000 mon.a (mon.0) 1027 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.260112+0000 mon.c (mon.1) 387 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.260430+0000 mon.a (mon.0) 1028 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.261997+0000 mon.c (mon.1) 388 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.263138+0000 mon.c (mon.1) 389 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.263497+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.264509+0000 mon.c (mon.1) 390 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.264864+0000 mon.a (mon.0) 1030 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.266394+0000 mon.c (mon.1) 391 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.349 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.267429+0000 mon.c (mon.1) 392 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.267740+0000 mon.a (mon.0) 1031 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.268692+0000 mon.c (mon.1) 393 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.081773+0000 mon.c (mon.1) 379 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.082129+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.087633+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]': finished 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.088838+0000 mon.c (mon.1) 380 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.089157+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.214891+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.241579+0000 mon.c (mon.1) 381 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.242491+0000 mon.c (mon.1) 382 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.350 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.242829+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.247926+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.iscsi"}]': finished 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.249417+0000 mon.c (mon.1) 383 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.250506+0000 mon.c (mon.1) 384 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.250822+0000 mon.a (mon.0) 1025 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.269009+0000 mon.a (mon.0) 1032 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.270466+0000 mon.c (mon.1) 394 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.271531+0000 mon.c (mon.1) 395 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.271850+0000 mon.a (mon.0) 1033 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.272778+0000 mon.c (mon.1) 396 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.273066+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.351 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.274118+0000 mon.c (mon.1) 397 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.274419+0000 mon.a (mon.0) 1035 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.275258+0000 mon.c (mon.1) 398 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.275514+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.387520+0000 mon.a (mon.0) 1037 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.388433+0000 mon.c (mon.1) 399 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.388686+0000 mon.a (mon.0) 1038 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.394004+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.394859+0000 mon.c (mon.1) 400 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.352 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.395094+0000 mon.a (mon.0) 1040 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.255908+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "client.nfs"}]': finished 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.257717+0000 mon.c (mon.1) 385 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.258868+0000 mon.c (mon.1) 386 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.259195+0000 mon.a (mon.0) 1027 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.260112+0000 mon.c (mon.1) 387 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.260430+0000 mon.a (mon.0) 1028 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.261997+0000 mon.c (mon.1) 388 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.263138+0000 mon.c (mon.1) 389 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.263497+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.264509+0000 mon.c (mon.1) 390 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.264864+0000 mon.a (mon.0) 1030 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.266394+0000 mon.c (mon.1) 391 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.267429+0000 mon.c (mon.1) 392 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.267740+0000 mon.a (mon.0) 1031 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.268692+0000 mon.c (mon.1) 393 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.399695+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.400434+0000 mon.c (mon.1) 401 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.400688+0000 mon.a (mon.0) 1042 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.401317+0000 mon.c (mon.1) 402 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.401529+0000 mon.a (mon.0) 1043 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.404777+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.408444+0000 mon.c (mon.1) 403 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.408680+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.409237+0000 mon.c (mon.1) 404 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.409432+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.354 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.412699+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.415771+0000 mon.c (mon.1) 405 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.415988+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.416542+0000 mon.c (mon.1) 406 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.416723+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.419870+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.423139+0000 mon.c (mon.1) 407 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.269009+0000 mon.a (mon.0) 1032 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.270466+0000 mon.c (mon.1) 394 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.271531+0000 mon.c (mon.1) 395 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.271850+0000 mon.a (mon.0) 1033 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon"}]: dispatch 2024-02-10T09:45:16.355 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.272778+0000 mon.c (mon.1) 396 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.273066+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.274118+0000 mon.c (mon.1) 397 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.274419+0000 mon.a (mon.0) 1035 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "global"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.275258+0000 mon.c (mon.1) 398 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.275514+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.387520+0000 mon.a (mon.0) 1037 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.388433+0000 mon.c (mon.1) 399 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.388686+0000 mon.a (mon.0) 1038 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.394004+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.394859+0000 mon.c (mon.1) 400 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.395094+0000 mon.a (mon.0) 1040 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.423341+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.426582+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.429521+0000 mon.c (mon.1) 408 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.429739+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.430278+0000 mon.c (mon.1) 409 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.430461+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.430982+0000 mon.c (mon.1) 410 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.431158+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.431673+0000 mon.c (mon.1) 411 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.431848+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.432369+0000 mon.c (mon.1) 412 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.432535+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.435851+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:45:16.357 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.439008+0000 mon.c (mon.1) 413 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.439244+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.441931+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.444156+0000 mon.c (mon.1) 414 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.445852+0000 mon.c (mon.1) 415 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.446050+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.448428+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.454594+0000 mon.c (mon.1) 416 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.456234+0000 mon.c (mon.1) 417 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.456442+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.358 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[35780]: audit 2024-02-10T09:45:15.458958+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.399695+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.400434+0000 mon.c (mon.1) 401 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.400688+0000 mon.a (mon.0) 1042 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.401317+0000 mon.c (mon.1) 402 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.401529+0000 mon.a (mon.0) 1043 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.404777+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.408444+0000 mon.c (mon.1) 403 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.408680+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:45:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.409237+0000 mon.c (mon.1) 404 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.409432+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.412699+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.415771+0000 mon.c (mon.1) 405 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.415988+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.416542+0000 mon.c (mon.1) 406 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.416723+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.419870+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.423139+0000 mon.c (mon.1) 407 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.423341+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.426582+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.429521+0000 mon.c (mon.1) 408 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.429739+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.430278+0000 mon.c (mon.1) 409 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.430461+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.430982+0000 mon.c (mon.1) 410 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.431158+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.431673+0000 mon.c (mon.1) 411 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.431848+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.432369+0000 mon.c (mon.1) 412 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.432535+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.435851+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.439008+0000 mon.c (mon.1) 413 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.439244+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.441931+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.444156+0000 mon.c (mon.1) 414 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.445852+0000 mon.c (mon.1) 415 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.446050+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.448428+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:16.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.454594+0000 mon.c (mon.1) 416 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:16.362 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.456234+0000 mon.c (mon.1) 417 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.362 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.456442+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:16.362 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:15 smithi153 bash[39117]: audit 2024-02-10T09:45:15.458958+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:17.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.028873+0000 mgr.y (mgr.34736) 115 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:45:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.054401+0000 mgr.y (mgr.34736) 116 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:45:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.069709+0000 mgr.y (mgr.34736) 117 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:45:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.081425+0000 mgr.y (mgr.34736) 118 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:45:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.242252+0000 mgr.y (mgr.34736) 119 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:45:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.250196+0000 mgr.y (mgr.34736) 120 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:45:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.258561+0000 mgr.y (mgr.34736) 121 : cephadm [INF] Upgrade: Setting container_image for all node-exporter 2024-02-10T09:45:17.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.262860+0000 mgr.y (mgr.34736) 122 : cephadm [INF] Upgrade: Setting container_image for all prometheus 2024-02-10T09:45:17.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.267171+0000 mgr.y (mgr.34736) 123 : cephadm [INF] Upgrade: Setting container_image for all alertmanager 2024-02-10T09:45:17.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.271257+0000 mgr.y (mgr.34736) 124 : cephadm [INF] Upgrade: Setting container_image for all grafana 2024-02-10T09:45:17.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.273860+0000 mgr.y (mgr.34736) 125 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:45:17.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[35780]: cephadm 2024-02-10T09:45:15.438629+0000 mgr.y (mgr.34736) 126 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.028873+0000 mgr.y (mgr.34736) 115 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.054401+0000 mgr.y (mgr.34736) 116 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.069709+0000 mgr.y (mgr.34736) 117 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.081425+0000 mgr.y (mgr.34736) 118 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.242252+0000 mgr.y (mgr.34736) 119 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.250196+0000 mgr.y (mgr.34736) 120 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:45:17.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.258561+0000 mgr.y (mgr.34736) 121 : cephadm [INF] Upgrade: Setting container_image for all node-exporter 2024-02-10T09:45:17.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.262860+0000 mgr.y (mgr.34736) 122 : cephadm [INF] Upgrade: Setting container_image for all prometheus 2024-02-10T09:45:17.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.267171+0000 mgr.y (mgr.34736) 123 : cephadm [INF] Upgrade: Setting container_image for all alertmanager 2024-02-10T09:45:17.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.271257+0000 mgr.y (mgr.34736) 124 : cephadm [INF] Upgrade: Setting container_image for all grafana 2024-02-10T09:45:17.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.273860+0000 mgr.y (mgr.34736) 125 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:45:17.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:16 smithi153 bash[39117]: cephadm 2024-02-10T09:45:15.438629+0000 mgr.y (mgr.34736) 126 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:45:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:17 smithi153 bash[35780]: cluster 2024-02-10T09:45:16.871645+0000 mgr.y (mgr.34736) 127 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:18.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:17 smithi153 bash[39117]: cluster 2024-02-10T09:45:16.871645+0000 mgr.y (mgr.34736) 127 : cluster [DBG] pgmap v66: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.601554+0000 mon.c (mon.1) 418 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:45:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.601853+0000 mon.c (mon.1) 419 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.602048+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:45:19.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.602092+0000 mon.c (mon.1) 420 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.602377+0000 mon.c (mon.1) 421 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.602669+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.602989+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.603299+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.795334+0000 mon.c (mon.1) 422 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}, {\"id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558301.5711062, \"finished_at\": 1707558315.4388149, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.601554+0000 mon.c (mon.1) 418 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.601853+0000 mon.c (mon.1) 419 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.602048+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:45:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.602092+0000 mon.c (mon.1) 420 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.602377+0000 mon.c (mon.1) 421 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.602669+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.602989+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.603299+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.795334+0000 mon.c (mon.1) 422 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}, {\"id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558301.5711062, \"finished_at\": 1707558315.4388149, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.795785+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}, {\"id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558301.5711062, \"finished_at\": 1707558315.4388149, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[35780]: audit 2024-02-10T09:45:18.800399+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}, {\"id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558301.5711062, \"finished_at\": 1707558315.4388149, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.795785+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}, {\"id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558301.5711062, \"finished_at\": 1707558315.4388149, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]: dispatch 2024-02-10T09:45:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:18 smithi153 bash[39117]: audit 2024-02-10T09:45:18.800399+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/progress/completed","val":"{\"events\": [{\"id\": \"7f69089c-5744-4376-aea0-4e49b8bf39bd\", \"message\": \"Rebalancing after osd.4 marked in\", \"refs\": [[\"osd\", 4]], \"started_at\": 1707557482.016496, \"finished_at\": 1707558198.7320318, \"add_to_ceph_s:\": 1707556818.7739785}, {\"id\": \"55b81bdf-cea8-4ef9-bfdb-b456cfd94f3d\", \"message\": \"Rebalancing after osd.5 marked in\", \"refs\": [[\"osd\", 5]], \"started_at\": 1707557482.0165048, \"finished_at\": 1707558198.7320364, \"add_to_ceph_s:\": 1707556847.0407863}, {\"id\": \"3f1e5b4b-3f96-4ce3-999c-743fb95a8842\", \"message\": \"Rebalancing after osd.6 marked in\", \"refs\": [[\"osd\", 6]], \"started_at\": 1707557482.0165062, \"finished_at\": 1707558198.7320378, \"add_to_ceph_s:\": 1707556865.7608364}, {\"id\": \"11e7f88d-2f9b-405e-be1a-dec11a261f91\", \"message\": \"Rebalancing after osd.7 marked in\", \"refs\": [[\"osd\", 7]], \"started_at\": 1707557482.0165074, \"finished_at\": 1707558198.7320392, \"add_to_ceph_s:\": 1707556879.842647}, {\"id\": \"041cb5ab-e230-402d-9186-c65545e7248c\", \"message\": \"Updating prometheus deployment (+1 -> 1)\", \"refs\": [], \"started_at\": 1707557508.3018928, \"finished_at\": 1707558198.7320402, \"add_to_ceph_s:\": 1707557486.3420982}, {\"id\": \"4537bccd-d371-4360-b72f-fdc86ea5abca\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558206.5345218, \"finished_at\": 1707558256.974483, \"add_to_ceph_s:\": true}, {\"id\": \"16dc7a59-1fe3-4c28-b6b4-20f2e897f08c\", \"message\": \"Upgrade to 16.2.14-521-g0e714d9a\", \"refs\": [], \"started_at\": 1707558301.5711062, \"finished_at\": 1707558315.4388149, \"add_to_ceph_s:\": true}], \"version\": 2, \"compat_version\": 2}"}]': finished 2024-02-10T09:45:20.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:19 smithi153 bash[35780]: cluster 2024-02-10T09:45:18.872044+0000 mgr.y (mgr.34736) 128 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:20.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:19 smithi153 bash[39117]: cluster 2024-02-10T09:45:18.872044+0000 mgr.y (mgr.34736) 128 : cluster [DBG] pgmap v67: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:20 smithi153 bash[35780]: cluster 2024-02-10T09:45:20.872756+0000 mgr.y (mgr.34736) 129 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:21.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:20 smithi153 bash[39117]: cluster 2024-02-10T09:45:20.872756+0000 mgr.y (mgr.34736) 129 : cluster [DBG] pgmap v68: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:24.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:45:24.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:23 smithi153 bash[39117]: cluster 2024-02-10T09:45:22.873248+0000 mgr.y (mgr.34736) 130 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:24.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:23 smithi153 bash[35780]: cluster 2024-02-10T09:45:22.873248+0000 mgr.y (mgr.34736) 130 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:24] "GET /metrics HTTP/1.1" 200 150274 "" "Prometheus/2.33.4" 2024-02-10T09:45:26.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:25 smithi153 bash[35780]: cluster 2024-02-10T09:45:24.874132+0000 mgr.y (mgr.34736) 131 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:26.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:25 smithi153 bash[39117]: cluster 2024-02-10T09:45:24.874132+0000 mgr.y (mgr.34736) 131 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:28.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:27 smithi153 bash[35780]: cluster 2024-02-10T09:45:26.874628+0000 mgr.y (mgr.34736) 132 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:28.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:27 smithi153 bash[39117]: cluster 2024-02-10T09:45:26.874628+0000 mgr.y (mgr.34736) 132 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:30.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:29 smithi153 bash[35780]: cluster 2024-02-10T09:45:28.875020+0000 mgr.y (mgr.34736) 133 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:30.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:29 smithi153 bash[39117]: cluster 2024-02-10T09:45:28.875020+0000 mgr.y (mgr.34736) 133 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:32.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:31 smithi153 bash[35780]: cluster 2024-02-10T09:45:30.875926+0000 mgr.y (mgr.34736) 134 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:32.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:31 smithi153 bash[39117]: cluster 2024-02-10T09:45:30.875926+0000 mgr.y (mgr.34736) 134 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:45:34.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:33 smithi153 bash[39117]: cluster 2024-02-10T09:45:32.876390+0000 mgr.y (mgr.34736) 135 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:34.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:33 smithi153 bash[35780]: cluster 2024-02-10T09:45:32.876390+0000 mgr.y (mgr.34736) 135 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:34] "GET /metrics HTTP/1.1" 200 150298 "" "Prometheus/2.33.4" 2024-02-10T09:45:36.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:35 smithi153 bash[35780]: cluster 2024-02-10T09:45:34.877079+0000 mgr.y (mgr.34736) 136 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:36.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:35 smithi153 bash[39117]: cluster 2024-02-10T09:45:34.877079+0000 mgr.y (mgr.34736) 136 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:37.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:45:38.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:37 smithi153 bash[39117]: cluster 2024-02-10T09:45:36.877478+0000 mgr.y (mgr.34736) 137 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:38.091 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:37 smithi153 bash[39117]: audit 2024-02-10T09:45:36.892649+0000 mgr.y (mgr.34736) 138 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:38.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:37 smithi153 bash[35780]: cluster 2024-02-10T09:45:36.877478+0000 mgr.y (mgr.34736) 137 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:38.091 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:37 smithi153 bash[35780]: audit 2024-02-10T09:45:36.892649+0000 mgr.y (mgr.34736) 138 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (13m) 96s ago 14m 26.0M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (88s) 24s ago 16m 49.2M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (118s) 24s ago 27m 386M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (9m) 96s ago 28m 427M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (28m) 96s ago 28m 204M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (29s) 24s ago 27m 24.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (27m) 96s ago 27m 185M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (106s) 96s ago 19m 11.9M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:45:39.365 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (101s) 24s ago 19m 14.2M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (26m) 96s ago 26m 43.8M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (26m) 96s ago 26m 44.4M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (26m) 96s ago 26m 43.1M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (25m) 96s ago 25m 46.0M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (25m) 24s ago 25m 47.8M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (25m) 24s ago 25m 48.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (24m) 24s ago 24m 46.9M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (24m) 24s ago 24m 45.6M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (13m) 24s ago 13m 66.0M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (16m) 96s ago 16m 35.9M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:45:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (16m) 24s ago 16m 37.6M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: cluster 2024-02-10T09:45:38.878008+0000 mgr.y (mgr.34736) 139 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.355083+0000 mon.c (mon.1) 423 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.355453+0000 mon.c (mon.1) 424 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.355776+0000 mon.c (mon.1) 425 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.356055+0000 mon.c (mon.1) 426 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.356317+0000 mon.c (mon.1) 427 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.356572+0000 mon.c (mon.1) 428 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.358211+0000 mon.c (mon.1) 429 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.358489+0000 mon.c (mon.1) 430 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.014 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.358775+0000 mon.c (mon.1) 431 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.359047+0000 mon.c (mon.1) 432 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[35780]: audit 2024-02-10T09:45:39.359305+0000 mon.c (mon.1) 433 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: cluster 2024-02-10T09:45:38.878008+0000 mgr.y (mgr.34736) 139 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.355083+0000 mon.c (mon.1) 423 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.355453+0000 mon.c (mon.1) 424 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.355776+0000 mon.c (mon.1) 425 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.356055+0000 mon.c (mon.1) 426 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.356317+0000 mon.c (mon.1) 427 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.356572+0000 mon.c (mon.1) 428 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.358211+0000 mon.c (mon.1) 429 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.358489+0000 mon.c (mon.1) 430 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.358775+0000 mon.c (mon.1) 431 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.359047+0000 mon.c (mon.1) 432 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:39 smithi153 bash[39117]: audit 2024-02-10T09:45:39.359305+0000 mon.c (mon.1) 433 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:40.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-02-10T09:45:41.304 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:40 smithi153 bash[35780]: audit 2024-02-10T09:45:39.353267+0000 mgr.y (mgr.34736) 140 : audit [DBG] from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:41.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:40 smithi153 bash[39117]: audit 2024-02-10T09:45:39.353267+0000 mgr.y (mgr.34736) 140 : audit [DBG] from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:42.088 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:45:42.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:41 smithi153 bash[35780]: cluster 2024-02-10T09:45:40.878585+0000 mgr.y (mgr.34736) 141 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:42.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:41 smithi153 bash[39117]: cluster 2024-02-10T09:45:40.878585+0000 mgr.y (mgr.34736) 141 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:42.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-02-10T09:45:43.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:42 smithi153 bash[35780]: audit 2024-02-10T09:45:42.077427+0000 mon.c (mon.1) 434 : audit [DBG] from='client.? 172.21.15.153:0/1908932531' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:43.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:42 smithi153 bash[39117]: audit 2024-02-10T09:45:42.077427+0000 mon.c (mon.1) 434 : audit [DBG] from='client.? 172.21.15.153:0/1908932531' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:45:44.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:43 smithi153 bash[35780]: cluster 2024-02-10T09:45:42.878976+0000 mgr.y (mgr.34736) 142 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:43 smithi153 bash[39117]: cluster 2024-02-10T09:45:42.878976+0000 mgr.y (mgr.34736) 142 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:44] "GET /metrics HTTP/1.1" 200 150298 "" "Prometheus/2.33.4" 2024-02-10T09:45:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.796546+0000 mon.c (mon.1) 435 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.796902+0000 mon.c (mon.1) 436 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.797194+0000 mon.c (mon.1) 437 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.797480+0000 mon.c (mon.1) 438 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.797743+0000 mon.c (mon.1) 439 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.798018+0000 mon.c (mon.1) 440 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.799573+0000 mon.c (mon.1) 441 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.799834+0000 mon.c (mon.1) 442 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.800131+0000 mon.c (mon.1) 443 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.800411+0000 mon.c (mon.1) 444 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[35780]: audit 2024-02-10T09:45:44.800680+0000 mon.c (mon.1) 445 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.796546+0000 mon.c (mon.1) 435 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.796902+0000 mon.c (mon.1) 436 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.797194+0000 mon.c (mon.1) 437 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.797480+0000 mon.c (mon.1) 438 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.797743+0000 mon.c (mon.1) 439 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.798018+0000 mon.c (mon.1) 440 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.799573+0000 mon.c (mon.1) 441 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.799834+0000 mon.c (mon.1) 442 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.800131+0000 mon.c (mon.1) 443 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.800411+0000 mon.c (mon.1) 444 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:45.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:44 smithi153 bash[39117]: audit 2024-02-10T09:45:44.800680+0000 mon.c (mon.1) 445 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:46.375 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:45 smithi153 bash[35780]: audit 2024-02-10T09:45:44.794663+0000 mgr.y (mgr.34736) 143 : audit [DBG] from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:46.375 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:45 smithi153 bash[35780]: cluster 2024-02-10T09:45:44.879649+0000 mgr.y (mgr.34736) 144 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:46.375 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:45 smithi153 bash[35780]: audit 2024-02-10T09:45:45.121039+0000 mgr.y (mgr.34736) 145 : audit [DBG] from='client.25096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mon", "hosts": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:46.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:45 smithi153 bash[39117]: audit 2024-02-10T09:45:44.794663+0000 mgr.y (mgr.34736) 143 : audit [DBG] from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:46.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:45 smithi153 bash[39117]: cluster 2024-02-10T09:45:44.879649+0000 mgr.y (mgr.34736) 144 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:46.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:45 smithi153 bash[39117]: audit 2024-02-10T09:45:45.121039+0000 mgr.y (mgr.34736) 145 : audit [DBG] from='client.25096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "mon", "hosts": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:47.006 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:47.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:45:48.132 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: cluster 2024-02-10T09:45:46.880109+0000 mgr.y (mgr.34736) 146 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:48.132 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.002374+0000 mon.c (mon.1) 446 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.002665+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.005854+0000 mon.a (mon.0) 1072 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.049067+0000 mon.c (mon.1) 447 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.050881+0000 mon.c (mon.1) 448 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.051100+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[39117]: audit 2024-02-10T09:45:47.054062+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: cluster 2024-02-10T09:45:46.880109+0000 mgr.y (mgr.34736) 146 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.002374+0000 mon.c (mon.1) 446 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.002665+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.005854+0000 mon.a (mon.0) 1072 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": null, \"target_digests\": null, \"target_version\": null, \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.049067+0000 mon.c (mon.1) 447 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:48.133 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.050881+0000 mon.c (mon.1) 448 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:48.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.051100+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch 2024-02-10T09:45:48.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:47 smithi153 bash[35780]: audit 2024-02-10T09:45:47.054062+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished 2024-02-10T09:45:49.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:48 smithi153 bash[39117]: cephadm 2024-02-10T09:45:47.001727+0000 mgr.y (mgr.34736) 147 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:49.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:48 smithi153 bash[39117]: cephadm 2024-02-10T09:45:47.058532+0000 mgr.y (mgr.34736) 148 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:49.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:48 smithi153 bash[35780]: cephadm 2024-02-10T09:45:47.001727+0000 mgr.y (mgr.34736) 147 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:49.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:48 smithi153 bash[35780]: cephadm 2024-02-10T09:45:47.058532+0000 mgr.y (mgr.34736) 148 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:45:50.169 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:45:50.306 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:49 smithi153 bash[35780]: cluster 2024-02-10T09:45:48.880499+0000 mgr.y (mgr.34736) 149 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:49 smithi153 bash[39117]: cluster 2024-02-10T09:45:48.880499+0000 mgr.y (mgr.34736) 149 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:50.811 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (13m) 107s ago 14m 26.0M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (99s) 35s ago 16m 49.2M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (2m) 35s ago 27m 386M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (9m) 107s ago 29m 427M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (29m) 107s ago 29m 204M 2048M 15.2.0 204a01f9b0b6 f3366761db0a 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (41s) 35s ago 27m 24.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (27m) 107s ago 27m 185M 2048M 15.2.0 204a01f9b0b6 58daa01e9308 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (117s) 107s ago 19m 11.9M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (112s) 35s ago 19m 14.2M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (26m) 107s ago 26m 43.8M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (26m) 107s ago 26m 44.4M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (26m) 107s ago 26m 43.1M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:45:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (25m) 107s ago 25m 46.0M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (25m) 35s ago 25m 47.8M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (25m) 35s ago 25m 48.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (25m) 35s ago 25m 46.9M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (24m) 35s ago 24m 45.6M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (14m) 35s ago 14m 66.0M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (16m) 107s ago 16m 35.9M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:45:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (16m) 35s ago 16m 37.6M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.325416+0000 mon.c (mon.1) 449 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.325793+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.328787+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.330242+0000 mon.c (mon.1) 450 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.331190+0000 mon.c (mon.1) 451 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.331414+0000 mon.a (mon.0) 1077 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.335522+0000 mon.a (mon.0) 1078 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.337533+0000 mon.c (mon.1) 452 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:51.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.325416+0000 mon.c (mon.1) 449 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.325793+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]: dispatch 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.328787+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1\", \"progress_id\": \"55355b0b-0b40-4b73-804f-a65a6e856fab\", \"target_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"target_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"target_version\": \"16.2.14-521-g0e714d9a\", \"fs_original_max_mds\": null, \"fs_original_allow_standby_replay\": null, \"error\": null, \"paused\": false, \"daemon_types\": [\"mon\"], \"hosts\": [\"smithi153\"], \"services\": null, \"total_count\": null, \"remaining_count\": null}"}]': finished 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.330242+0000 mon.c (mon.1) 450 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.331190+0000 mon.c (mon.1) 451 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.331414+0000 mon.a (mon.0) 1077 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]: dispatch 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.335522+0000 mon.a (mon.0) 1078 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "mon_mds_skip_sanity", "value": "1", "who": "mon"}]': finished 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.337533+0000 mon.c (mon.1) 452 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:51.151 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.338391+0000 mon.c (mon.1) 453 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.338654+0000 mon.a (mon.0) 1079 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.342734+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.344057+0000 mon.c (mon.1) 454 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.344766+0000 mon.c (mon.1) 455 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.800867+0000 mon.c (mon.1) 456 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.801223+0000 mon.c (mon.1) 457 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.801550+0000 mon.c (mon.1) 458 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.801864+0000 mon.c (mon.1) 459 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.802166+0000 mon.c (mon.1) 460 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.802452+0000 mon.c (mon.1) 461 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.804088+0000 mon.c (mon.1) 462 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:51.152 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.804395+0000 mon.c (mon.1) 463 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.804683+0000 mon.c (mon.1) 464 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.804975+0000 mon.c (mon.1) 465 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[35780]: audit 2024-02-10T09:45:50.805267+0000 mon.c (mon.1) 466 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.338391+0000 mon.c (mon.1) 453 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.338654+0000 mon.a (mon.0) 1079 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.342734+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mgr"}]': finished 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.344057+0000 mon.c (mon.1) 454 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.344766+0000 mon.c (mon.1) 455 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.800867+0000 mon.c (mon.1) 456 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.801223+0000 mon.c (mon.1) 457 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.801550+0000 mon.c (mon.1) 458 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.801864+0000 mon.c (mon.1) 459 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.153 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.802166+0000 mon.c (mon.1) 460 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.154 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.802452+0000 mon.c (mon.1) 461 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.154 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.804088+0000 mon.c (mon.1) 462 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:45:51.154 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.804395+0000 mon.c (mon.1) 463 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.154 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.804683+0000 mon.c (mon.1) 464 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.154 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.804975+0000 mon.c (mon.1) 465 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.154 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:50 smithi153 bash[39117]: audit 2024-02-10T09:45:50.805267+0000 mon.c (mon.1) 466 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2, 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 1 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:45:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 12, 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:45:51.211 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi153", 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:45:51.527 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:50.157381+0000 mgr.y (mgr.34736) 150 : audit [DBG] from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: cephadm 2024-02-10T09:45:50.329364+0000 mgr.y (mgr.34736) 151 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: cephadm 2024-02-10T09:45:50.329421+0000 mgr.y (mgr.34736) 152 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: cephadm 2024-02-10T09:45:50.338177+0000 mgr.y (mgr.34736) 153 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: cephadm 2024-02-10T09:45:50.345266+0000 mgr.y (mgr.34736) 154 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:50.476875+0000 mgr.y (mgr.34736) 155 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:51.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:50.799022+0000 mgr.y (mgr.34736) 156 : audit [DBG] from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: cluster 2024-02-10T09:45:50.881148+0000 mgr.y (mgr.34736) 157 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.209923+0000 mon.a (mon.0) 1081 : audit [DBG] from='client.? 172.21.15.153:0/827884536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.721839+0000 mon.c (mon.1) 467 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.a"}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.722167+0000 mon.a (mon.0) 1082 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.a"}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.728274+0000 mon.a (mon.0) 1083 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.a"}]': finished 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.729265+0000 mon.c (mon.1) 468 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.729817+0000 mon.c (mon.1) 469 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.730333+0000 mon.c (mon.1) 470 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[39117]: audit 2024-02-10T09:45:51.731221+0000 mon.c (mon.1) 471 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:50.157381+0000 mgr.y (mgr.34736) 150 : audit [DBG] from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: cephadm 2024-02-10T09:45:50.329364+0000 mgr.y (mgr.34736) 151 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: cephadm 2024-02-10T09:45:50.329421+0000 mgr.y (mgr.34736) 152 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:45:51.958 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: cephadm 2024-02-10T09:45:50.338177+0000 mgr.y (mgr.34736) 153 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: cephadm 2024-02-10T09:45:50.345266+0000 mgr.y (mgr.34736) 154 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:50.476875+0000 mgr.y (mgr.34736) 155 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:50.799022+0000 mgr.y (mgr.34736) 156 : audit [DBG] from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: cluster 2024-02-10T09:45:50.881148+0000 mgr.y (mgr.34736) 157 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.209923+0000 mon.a (mon.0) 1081 : audit [DBG] from='client.? 172.21.15.153:0/827884536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.721839+0000 mon.c (mon.1) 467 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.a"}]: dispatch 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.722167+0000 mon.a (mon.0) 1082 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.a"}]: dispatch 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.728274+0000 mon.a (mon.0) 1083 : audit [INF] from='mgr.34736 ' entity='mgr.y' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", "who": "mon.a"}]': finished 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.729265+0000 mon.c (mon.1) 468 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:45:51.959 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.729817+0000 mon.c (mon.1) 469 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:45:52.379 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.730333+0000 mon.c (mon.1) 470 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:45:52.379 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:51 smithi153 bash[35780]: audit 2024-02-10T09:45:51.731221+0000 mon.c (mon.1) 471 : audit [DBG] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:45:53.368 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:52 smithi153 bash[39117]: audit 2024-02-10T09:45:51.526436+0000 mgr.y (mgr.34736) 158 : audit [DBG] from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:52 smithi153 bash[39117]: cephadm 2024-02-10T09:45:51.721420+0000 mgr.y (mgr.34736) 159 : cephadm [INF] Upgrade: Updating mon.a 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:52 smithi153 bash[39117]: cephadm 2024-02-10T09:45:51.730859+0000 mgr.y (mgr.34736) 160 : cephadm [INF] Deploying daemon mon.a on smithi153 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:52 smithi153 bash[35780]: audit 2024-02-10T09:45:51.526436+0000 mgr.y (mgr.34736) 158 : audit [DBG] from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:52 smithi153 bash[35780]: cephadm 2024-02-10T09:45:51.721420+0000 mgr.y (mgr.34736) 159 : cephadm [INF] Upgrade: Updating mon.a 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:52 smithi153 bash[35780]: cephadm 2024-02-10T09:45:51.730859+0000 mgr.y (mgr.34736) 160 : cephadm [INF] Deploying daemon mon.a on smithi153 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:53 smithi153 systemd[1]: Stopping Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:53 smithi153 bash[35780]: debug 2024-02-10T09:45:53.158+0000 7f47f05ae700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:45:53.369 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:53 smithi153 bash[35780]: debug 2024-02-10T09:45:53.158+0000 7f47f05ae700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T09:45:54.036 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:53 smithi153 bash[71921]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.a 2024-02-10T09:45:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:45:54.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:54 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service: Succeeded. 2024-02-10T09:45:54.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:54 smithi153 systemd[1]: Stopped Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:45:54.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:54 smithi153 systemd[1]: Starting Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:45:54.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:45:54] "GET /metrics HTTP/1.1" 200 150299 "" "Prometheus/2.33.4" 2024-02-10T09:45:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.034+0000 7ff678e62880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:45:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.034+0000 7ff678e62880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-10T09:45:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.035+0000 7ff678e62880 0 pidfile_write: ignore empty --pid-file 2024-02-10T09:45:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.039+0000 7ff678e62880 0 load: jerasure load: lrc load: isa 2024-02-10T09:45:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: DB SUMMARY 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: CURRENT file: CURRENT 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 2924 Bytes 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 2, files: 000068.sst 000069.sst 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000066.log size: 3502559 ; 2024-02-10T09:45:55.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.create_if_missing: 0 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.env: 0x55cbe4f5c080 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.fs: Posix File System 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.info_log: 0x55cbe5e46260 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:45:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.db_log_dir: 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:45:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.write_buffer_manager: 0x55cbe6b38390 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.unordered_write: 0 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:45:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.row_cache: None 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.wal_filter: None 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T09:45:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:45:55.446 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Compression algorithms supported: 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kZSTD supported: 0 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.merge_operator: 2024-02-10T09:45:55.447 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cbe5debd28) 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cache_index_and_filter_blocks: 1 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: pin_top_level_index_and_filter: 1 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: index_type: 0 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: data_block_index_type: 0 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: index_shortening: 1 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: hash_index_allow_collision: 1 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: checksum: 1 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: no_block_cache: 0 2024-02-10T09:45:55.448 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_cache: 0x55cbe5e20f10 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_cache_name: BinnedLRUCache 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_cache_options: 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: capacity : 536870912 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: num_shard_bits : 4 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: strict_capacity_limit : 0 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: high_pri_pool_ratio: 0.000 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_cache_compressed: (nil) 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: persistent_cache: (nil) 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_size: 4096 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_size_deviation: 10 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_restart_interval: 16 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: index_block_restart_interval: 1 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: metadata_block_size: 4096 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: partition_filters: 0 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: use_delta_encoding: 1 2024-02-10T09:45:55.449 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: whole_key_filtering: 1 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: verify_compression: 0 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: read_amp_bytes_per_bit: 0 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: format_version: 2 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: enable_index_compression: 1 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: block_align: 0 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:45:55.451 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.num_levels: 7 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:45:55.452 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:45:55.453 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:45:55.454 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:45:55.455 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.ttl: 2592000 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.040+0000 7ff678e62880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.041+0000 7ff678e62880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 71, last_sequence is 23407, log_number is 66,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.041+0000 7ff678e62880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 66 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.041+0000 7ff678e62880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558355042981, "job": 1, "event": "recovery_started", "log_files": [66]} 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.041+0000 7ff678e62880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #66 mode 2 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.044+0000 7ff678e62880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.048+0000 7ff678e62880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558355050216, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 71, "file_size": 3493003, "table_properties": {"data_size": 3488293, "index_size": 3311, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 5633, "raw_average_key_size": 26, "raw_value_size": 3482245, "raw_average_value_size": 16503, "num_data_blocks": 103, "num_entries": 211, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707558355, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.048+0000 7ff678e62880 4 rocksdb: [version_set.cc:3826] Creating manifest 72 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.050+0000 7ff678e62880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558355052656, "job": 1, "event": "recovery_finished"} 2024-02-10T09:45:55.456 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.051+0000 7ff678e62880 4 rocksdb: DB pointer 0x55cbe5ee9800 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.052+0000 7ff65fc53700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.052+0000 7ff65fc53700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** DB Stats ** 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** Compaction Stats [default] ** 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: L0 1/0 3.33 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.457 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: L6 2/0 118.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Sum 3/0 121.72 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** Compaction Stats [default] ** 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Flush(GB): cumulative 0.003, interval 0.003 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Cumulative compaction: 0.00 GB write, 287.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Interval compaction: 0.00 GB write, 287.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:55.458 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** Compaction Stats [default] ** 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: L0 1/0 3.33 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: L6 2/0 118.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Sum 3/0 121.72 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** Compaction Stats [default] ** 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 783.6 0.00 0.00 1 0.004 0 0 2024-02-10T09:45:55.459 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Flush(GB): cumulative 0.003, interval 0.000 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Cumulative compaction: 0.00 GB write, 287.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.052+0000 7ff678e62880 0 starting mon.a rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.053+0000 7ff678e62880 1 mon.a@-1(???) e3 preinit fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 0 mon.a@-1(???).mds e1 new map 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 0 mon.a@-1(???).mds e1 print_map 2024-02-10T09:45:55.460 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: e1 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: enable_multiple, ever_enabled_multiple: 0,0 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: legacy client fscid: -1 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: No filesystems configured 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 0 mon.a@-1(???).osd e72 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 0 mon.a@-1(???).osd e72 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 0 mon.a@-1(???).osd e72 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 0 mon.a@-1(???).osd e72 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:45:51.526436+0000 mgr.y (mgr.34736) 158 : audit [DBG] from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:45:51.721420+0000 mgr.y (mgr.34736) 159 : cephadm [INF] Upgrade: Updating mon.a 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:45:51.730859+0000 mgr.y (mgr.34736) 160 : cephadm [INF] Deploying daemon mon.a on smithi153 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:45:55.054+0000 7ff678e62880 1 mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 bash[72113]: 029c82f28e3c5b421ecaf06c0e0ebbae51f5ee30234a8288113dcce0eb6fee9b 2024-02-10T09:45:55.461 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:55 smithi153 systemd[1]: Started Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:45:56.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ignoring --setuser ceph since I am not root 2024-02-10T09:45:56.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ignoring --setgroup ceph since I am not root 2024-02-10T09:45:56.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:56.549+0000 7f30ce634700 1 -- 172.21.15.204:0/1309559176 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56472a20c4e0 con 0x56472a1e2c00 2024-02-10T09:45:56.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:56.672+0000 7f30de812680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:45:56.832 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:56.753+0000 7f30de812680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:45:56.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ignoring --setuser ceph since I am not root 2024-02-10T09:45:56.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ignoring --setgroup ceph since I am not root 2024-02-10T09:45:56.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:56.550+0000 7fc5269f9700 1 -- 172.21.15.153:0/2236654238 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5588b58664e0 con 0x5588b583d400 2024-02-10T09:45:56.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:56.671+0000 7fc536bd7680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T09:45:56.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:56.752+0000 7fc536bd7680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T09:45:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:54.882189+0000 mgr.y (mgr.34736) 162 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:45:55.160710+0000 mon.c (mon.1) 472 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"service_name\": \"mon\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \ 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:54.882189+0000 mgr.y (mgr.34736) 162 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 6.2 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: "2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:51.730765Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.464538+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.478715+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.483389+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} removed_ranks: {} 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.483928+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.483967+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.484981+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e39: y(active, since 2m), standbys: x 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.501226+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e40: y(active, since 2m), standbys: x 2024-02-10T09:45:56.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:56.502016+0000 mon.a (mon.0) 8 : cluster [INF] overall HEALTH_OK 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:45:56.509457+0000 mon.a (mon.0) 9 : audit [INF] from='mgr.34736 ' entity='' 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: audit 2024-02-10T09:45:55.160710+0000 mon.c (mon.1) 472 : audit [INF] from='mgr.34736 172.21.15.153:0/724399821' entity='mgr.y' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi153","val":"{\"daemons\": {\"mon.a\": {\"daemon_type\": \"mon\", \"daemon_id\": \"a\", \"service_name\": \"mon\", \"daemon_name\": \"mon.a\", \"hostname\": \"smithi153\", \"status\": 2, \"status_desc\": \"starting\", \"is_active\": false, \"ports\": []}, \"mgr.y\": {\"daemon_type\": \"mgr\", \"daemon_id\": \"y\", \"service_name\": \"mgr\", \"daemon_name\": \"mgr.y\", \"hostname\": \"smithi153\", \"container_id\": \"a9598abcef48\", \"container_image_id\": \"1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5\", \"container_image_name\": \"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\", \"container_image_digests\": [\"quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc\"], \"memory_usage\": 448266240, \"cpu_percentage\": \"4.40%\", \"version\": \"16.2.14-521-g0e714d9a\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [8443, 9283], \"last_refresh\": \"2024-02-10T09:44:02.966204Z\", \"created\": \"2024-02-10T09:16:47.620498Z\", \"started\": \"2024-02-10T09:35:54.615301Z\"}, \"mon.c\": {\"daemon_type\": \"mon\", \"daemon_id\": \"c\", \"daemon_name\": \"mon.c\", \"hostname\": \"smithi153\", \"container_id\": \"58daa01e9308\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 194825420, \"cpu_percentage\": \"0.74%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966272Z\", \"created\": \"2024-02-10T09:18:01.850164Z\", \"started\": \"2024-02-10T09:18:01.883088Z\"}, \"osd.0\": {\"daemon_type\": \"osd\", \"daemon_id\": \"0\", \"daemon_name\": \"osd.0\", \"hostname\": \"smithi153\", \"container_id\": \"90f82f6e81ca\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46001029, \"cpu_percentage\": \"0.50%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966335Z\", \"created\": \"2024-02-10T09:18:52.144251Z\", \"started\": \"2024-02-10T09:19:13.914993Z\"}, \"osd.1\": {\"daemon_type\": \"osd\", \"daemon_id\": \"1\", \"daemon_name\": \"osd.1\", \"hostname\": \"smithi153\", \"container_id\": \"d213cd3c6620\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 46598717, \"cpu_percentage\": \"0.51%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966398Z\", \"created\": \"2024-02-10T09:19:12.921872Z\", \"started\": \"2024-02-10T09:19:35.395036Z\"}, \"osd.2\": {\"daemon_type\": \"osd\", \"daemon_id\": \"2\", \"daemon_name\": \"osd.2\", \"hostname\": \"smithi153\", \"container_id\": \"fd3d7b02f03d\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 45214597, \"cpu_percentage\": \"0.40%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966461Z\", \"created\": \"2024-02-10T09:19:36.993433Z\", \"started\": \"2024-02-10T09:19:39.009093Z\"}, \"osd.3\": {\"daemon_type\": \"osd\", \"daemon_id\": \"3\", \"daemon_name\": \"osd.3\", \"hostname\": \"smithi153\", \"container_id\": \"90119323f9b5\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 48328867, \"cpu_percentage\": \"0.52%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966522Z\", \"created\": \"2024-02-10T09:19:55.310100Z\", \"started\": \"2024-02-10T09:19:57.924701Z\"}, \"node-exporter.a\": {\"daemon_type\": \"node-exporter\", \"daemon_id\": \"a\", \"service_name\": \"node-exporter\", \"daemon_name\": \"node-exporter.a\", \"hostname\": \"smithi153\", \"container_id\": \"086dc26b8c2c\", \"container_image_id\": \"1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26\", \"container_image_name\": \"quay.io/prometheus/node-exporter:v1.3.1\", \"container_image_digests\": [\"quay.io/prometheus/node-exporter@sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd\", \"quay.io/prometheus/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d\"], \"memory_usage\": 12530483, \"cpu_percentage\": \"1.57%\", \"version\": \"1.3.1\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"ports\": [], \"last_refresh\": \"2024-02-10T09:44:02.966583Z\", \"created\": \"2024-02-10T09:26:14.154248Z\", \"started\": \"2024-02-10T09:43:53.124365Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"daemon_type\": \"rgw\", \"daemon_id\": \"r.z.smithi153.zyveqt\", \"daemon_name\": \"rgw.r.z.smithi153.zyveqt\", \"hostname\": \"smithi153\", \"container_id\": \"4417ab8bf66a\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"container_image_digests\": [\"docker.io/ceph/ceph@sha256:54b064d6a616656497df507997f8cd47f96ec7cc137f18b69f74ea03a1467c60\", \"docker.io/ceph/ceph@sha256:1cc9b824e1b076cdff52a9aa3f0cc8557d879fb2fbbba0cafed970aca59a3992\"], \"memory_usage\": 37717278, \"cpu_percentage\": \"0.14%\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": false, \"last_refresh\": \"2024-02-10T09:44:02.966644Z\", \"created\": \"2024-02-10T09:29:02.635166Z\", \"started\": \"2024-02-10T09:29:02.666766Z\"}, \"alertmanager.a\": {\"daemon_type\": \"alertmanager\", \"daemon_id\": \"a\", \"service_name\": \"alertmanager\", \"daemon_name\": \"alertmanager.a\", \"hostname\": \"smithi153\", \"container_id\": \"a318f98f94b4\", \"container_image_id\": \"ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4\", \"container_image_name\": \"quay.io/prometheus/alertmanager:v0.23.0\", \"container_image_digests\": [\"quay.io/prometheus/alertmanager@sha256:3915fd90525daa23260082194e3ca8a75c51af5d436b64eeff16f0c7e54ef829\", \"quay.io/prometheus/alertmanager@sha256:9ab73a421b65b80be072f96a88df756fc5b52a1bc8d983537b8ec5be8b624c5a\"], \"memory_usage\": 27325890, \"cpu_percentage\": \"0.10%\", \"version\": \"0.23.0\", \"status\": 1, \"status_desc\": \"running\", \"is_active\": true, \"ports\": [9093, 9094], \"last_refresh\": \"2024-02-10T09:44:02.966706Z\", \"created\": \"2024-02-10T09:31:09.057848Z\", \"started\": \"2024-02-10T09:31:59.810371Z\"}}, \"devices\": [{\"ceph_device\": null, \"rejected_reasons\": [\"Insufficient space (<5GB) on vgs\", \"LVM detected\", \"locked\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945503Z\", \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"679b994c-2b1f-4f4d-9b26-64a3d365a094\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"id2xej-7BEC-Pl3s-vxQY-r2pI-HhvD-H3N6Cf\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b6e5a960-d7e8-4e2c-b913-6848f7f0de36\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"iFVutn-tVqA-jAED-Up2S-3Sc0-ssIr-87OxrI\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"fa508d13-ae9b-4d62-b567-1be1a9d46d5c\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"f5mOma-gpQb-cie8-mRuN-Hhyo-5yzu-k9XXpU\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"8351789c-a971-44a1-aa2f-b16e87fb72a8\", \"cluster_fsid\": \"09b43244-c7f5-11ee-95b8-87774f69a715\", \"block_uuid\": \"vmb8ZX-coZa-cR3I-rOU8-YGvX-2PLf-YdbFq1\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300MV400BGN\", \"lsm_data\": {}}, {\"ceph_device\": null, \"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"created\": \"2024-02-10T09:30:23.945515Z\", \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM_Z1W5XFQY\", \"lsm_data\": {}}], \"osdspec_previews\": [], \"osdspec_last_applied\": {}, \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2024-02-10T09:31:54.173463Z\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2024-02-10T09:35:41.581071Z\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:07.111656Z\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:30.146058Z\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:33.554255Z\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2024-02-10T09:19:53.783530Z\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:43:50.559502Z\"}, \"rgw.r.z.smithi153.zyveqt\": {\"deps\": [], \"last_config\": \"2024-02-10T09:29:00.493650Z\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2024-02-10T09:31:55.481137Z\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2024-02-10T09:45:51.730765Z\"}}, \"last_device_update\": \"2024-02-10T09:20:01.571358Z\", \"networks_and_interfaces\": {}, \"last_host_check\": \"2024-02-10T09:40:01.482220Z\", \"last_client_files\": {}, \"scheduled_daemon_actions\": {}}"}]: dispatch 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.464538+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.478715+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.483389+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} removed_ranks: {} 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.483928+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.483967+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.484981+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e39: y(active, since 2m), standbys: x 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.501226+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e40: y(active, since 2m), standbys: x 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: cluster 2024-02-10T09:45:56.502016+0000 mon.a (mon.0) 8 : cluster [INF] overall HEALTH_OK 2024-02-10T09:45:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:56 smithi153 bash[39117]: audit 2024-02-10T09:45:56.509457+0000 mon.a (mon.0) 9 : audit [INF] from='mgr.34736 ' entity='' 2024-02-10T09:45:57.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:57.113+0000 7fc536bd7680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:45:57.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:57.116+0000 7f30de812680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T09:45:58.156 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:57.857+0000 7fc536bd7680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:45:58.156 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:57.942+0000 7fc536bd7680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:45:58.171 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:57.869+0000 7f30de812680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T09:45:58.171 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:57.955+0000 7f30de812680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T09:45:58.581 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:58.169+0000 7f30de812680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:45:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:45:58 smithi153 bash[39117]: cluster 2024-02-10T09:45:57.502344+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e41: y(active, since 2m), standbys: x 2024-02-10T09:45:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:45:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:45:57.502344+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e41: y(active, since 2m), standbys: x 2024-02-10T09:45:58.940 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:58.154+0000 7fc536bd7680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T09:45:59.401 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.099+0000 7fc536bd7680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:45:59.401 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.228+0000 7fc536bd7680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:45:59.401 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.308+0000 7fc536bd7680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:45:59.414 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.114+0000 7f30de812680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T09:45:59.414 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.242+0000 7f30de812680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:45:59.414 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.322+0000 7f30de812680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T09:45:59.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.399+0000 7fc536bd7680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:45:59.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.600+0000 7fc536bd7680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:45:59.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.413+0000 7f30de812680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T09:45:59.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.611+0000 7f30de812680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T09:46:00.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.881+0000 7fc536bd7680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:46:00.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:45:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:45:59.980+0000 7fc536bd7680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:46:00.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.893+0000 7f30de812680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T09:46:00.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:45:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:45:59.992+0000 7f30de812680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T09:46:01.070 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:00.711+0000 7fc536bd7680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:46:01.070 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:00.795+0000 7fc536bd7680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:46:01.070 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:00.889+0000 7fc536bd7680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:46:01.070 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:01.067+0000 7fc536bd7680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:46:01.078 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:00.723+0000 7f30de812680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T09:46:01.078 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:00.806+0000 7f30de812680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T09:46:01.078 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:00.899+0000 7f30de812680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T09:46:01.078 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:01.077+0000 7f30de812680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T09:46:01.601 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:01.330+0000 7fc536bd7680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:46:01.601 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:01.463+0000 7fc536bd7680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:46:01.606 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:01.338+0000 7f30de812680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T09:46:01.606 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:01.471+0000 7f30de812680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T09:46:01.862 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:01.599+0000 7fc536bd7680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:46:01.862 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:01.773+0000 7fc536bd7680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:46:01.868 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:01.605+0000 7f30de812680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T09:46:01.868 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:01.780+0000 7f30de812680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T09:46:02.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:01.860+0000 7fc536bd7680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:46:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:01.875149+0000 mon.a (mon.0) 11 : cluster [INF] Active manager daemon y restarted 2024-02-10T09:46:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:01.875415+0000 mon.a (mon.0) 12 : cluster [INF] Activating manager daemon y 2024-02-10T09:46:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:01.883429+0000 mon.a (mon.0) 13 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T09:46:02.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:01.883818+0000 mon.a (mon.0) 14 : cluster [DBG] mgrmap e42: y(active, starting, since 0.00847322s), standbys: x 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:01.884387+0000 mon.a (mon.0) 15 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:01.884423+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:01.884457+0000 mon.a (mon.0) 16 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:01.885155+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:01.887122+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:01.887883+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: cluster 2024-02-10T09:46:01.875149+0000 mon.a (mon.0) 11 : cluster [INF] Active manager daemon y restarted 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: cluster 2024-02-10T09:46:01.875415+0000 mon.a (mon.0) 12 : cluster [INF] Activating manager daemon y 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: cluster 2024-02-10T09:46:01.883429+0000 mon.a (mon.0) 13 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: cluster 2024-02-10T09:46:01.883818+0000 mon.a (mon.0) 14 : cluster [DBG] mgrmap e42: y(active, starting, since 0.00847322s), standbys: x 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: cluster 2024-02-10T09:46:01.884387+0000 mon.a (mon.0) 15 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: audit 2024-02-10T09:46:01.884423+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T09:46:02.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: cluster 2024-02-10T09:46:01.884457+0000 mon.a (mon.0) 16 : cluster [DBG] Standby manager daemon x started 2024-02-10T09:46:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: audit 2024-02-10T09:46:01.885155+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:46:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: audit 2024-02-10T09:46:01.887122+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T09:46:02.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:01 smithi153 bash[39117]: audit 2024-02-10T09:46:01.887883+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.204:0/1200204995' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:46:02.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: debug 2024-02-10T09:46:01.866+0000 7f30de812680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T09:46:02.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: [10/Feb/2024:09:46:01] ENGINE Bus STARTING 2024-02-10T09:46:02.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: CherryPy Checker: 2024-02-10T09:46:02.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: The Application mounted at '' has an empty config. 2024-02-10T09:46:02.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: 2024-02-10T09:46:02.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: [10/Feb/2024:09:46:01] ENGINE Serving on http://:::9283 2024-02-10T09:46:02.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: [10/Feb/2024:09:46:01] ENGINE Bus STARTED 2024-02-10T09:46:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:46:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:03 smithi153 bash[39117]: cluster 2024-02-10T09:46:02.883553+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e43: y(active, starting, since 1.0082s), standbys: x 2024-02-10T09:46:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:03 smithi153 bash[39117]: audit 2024-02-10T09:46:02.887043+0000 mon.c (mon.1) 473 : audit [DBG] from='mgr.? 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:46:04.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:03 smithi153 bash[39117]: audit 2024-02-10T09:46:02.887221+0000 mon.c (mon.1) 474 : audit [DBG] from='mgr.? 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:46:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:02.883553+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e43: y(active, starting, since 1.0082s), standbys: x 2024-02-10T09:46:04.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:02.887043+0000 mon.c (mon.1) 473 : audit [DBG] from='mgr.? 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T09:46:04.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:02.887221+0000 mon.c (mon.1) 474 : audit [DBG] from='mgr.? 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:07.975+0000 7fc4ea3a1700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:46:08] ENGINE Bus STARTING 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [80B blob data] 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [80B blob data] 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: CherryPy Checker: 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.901718+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:46:08.205 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.901946+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.902135+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.902456+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.902780+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.903031+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.903310+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.903645+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.903938+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.904186+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.904432+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.904680+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.904866+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.905341+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.906163+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.907400+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:46:08.206 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.907758+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.908068+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.908348+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.908648+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.908959+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.909240+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.909514+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.909805+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.910130+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.910414+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.910695+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.910995+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.911278+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:46:08.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.901718+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.901946+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.902135+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.902456+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.902780+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.903031+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.903310+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.903645+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.903938+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.904186+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.904432+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.904680+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:46:08.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.904866+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.905341+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.906163+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.907400+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/config_checks"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.907758+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_crt"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.908068+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/grafana_key"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.911558+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.911858+0000 mon.a (mon.0) 48 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.912197+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.912504+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:46:08.209 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.912790+0000 mon.a (mon.0) 51 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.913077+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.913375+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.913692+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.914020+0000 mon.a (mon.0) 55 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.914305+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.914581+0000 mon.a (mon.0) 57 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:07.914852+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:07.917256+0000 mon.a (mon.0) 59 : cluster [INF] Manager daemon y is now available 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.908348+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi153"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.908648+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi204"}]: dispatch 2024-02-10T09:46:08.210 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.908959+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.909240+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.909514+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_crt"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.909805+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/smithi204/grafana_key"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.910130+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.alertmanager"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.910414+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.grafana"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.910695+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.mgr"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.910995+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.node-exporter"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.911278+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.prometheus"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.911558+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/spec.rgw.r.z"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.911858+0000 mon.a (mon.0) 48 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.912197+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.912504+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:46:08.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.912790+0000 mon.a (mon.0) 51 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.913077+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.913375+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.913692+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.914020+0000 mon.a (mon.0) 55 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.914305+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/prometheus/health_history"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.914581+0000 mon.a (mon.0) 57 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: audit 2024-02-10T09:46:07.914852+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2024-02-10T09:46:08.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:07 smithi153 bash[39117]: cluster 2024-02-10T09:46:07.917256+0000 mon.a (mon.0) 59 : cluster [INF] Manager daemon y is now available 2024-02-10T09:46:08.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: The Application mounted at '' has an empty config. 2024-02-10T09:46:08.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:46:08.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:46:08] ENGINE Serving on http://:::9283 2024-02-10T09:46:08.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:46:08] ENGINE Bus STARTED 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: cluster 2024-02-10T09:46:07.975125+0000 mon.a (mon.0) 60 : cluster [DBG] mgrmap e44: y(active, since 6s), standbys: x 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: cluster 2024-02-10T09:46:07.983989+0000 mgr.y (mgr.25138) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: audit 2024-02-10T09:46:08.031036+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: audit 2024-02-10T09:46:08.032777+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: audit 2024-02-10T09:46:08.077682+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: audit 2024-02-10T09:46:08.081226+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:46:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: audit 2024-02-10T09:46:08.303067+0000 mon.a (mon.0) 65 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:08 smithi153 bash[39117]: audit 2024-02-10T09:46:08.313297+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:07.975125+0000 mon.a (mon.0) 60 : cluster [DBG] mgrmap e44: y(active, since 6s), standbys: x 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:07.983989+0000 mgr.y (mgr.25138) 1 : cluster [DBG] pgmap v3: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:08.031036+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:08.032777+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:08.077682+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:08.081226+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:08.303067+0000 mon.a (mon.0) 65 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:46:09.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:08.313297+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:46:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:10 smithi153 bash[39117]: cluster 2024-02-10T09:46:09.903586+0000 mgr.y (mgr.25138) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:10 smithi153 bash[39117]: cluster 2024-02-10T09:46:09.986827+0000 mon.a (mon.0) 67 : cluster [DBG] mgrmap e45: y(active, since 8s), standbys: x 2024-02-10T09:46:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:09.903586+0000 mgr.y (mgr.25138) 2 : cluster [DBG] pgmap v4: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:09.986827+0000 mon.a (mon.0) 67 : cluster [DBG] mgrmap e45: y(active, since 8s), standbys: x 2024-02-10T09:46:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: cluster 2024-02-10T09:46:11.903925+0000 mgr.y (mgr.25138) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.322426+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.592998+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.593356+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.593610+0000 mon.a (mon.0) 71 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.593892+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.594148+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.594387+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.594628+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.594877+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.595132+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:12.982419+0000 mon.a (mon.0) 78 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.260719+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.261124+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.261461+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.261799+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.262121+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.262441+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.262714+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.263006+0000 mon.a (mon.0) 86 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.263302+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.267405+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.272643+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.274740+0000 mon.a (mon.0) 90 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.275756+0000 mon.a (mon.0) 91 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:46:13.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:13 smithi153 bash[39117]: audit 2024-02-10T09:46:13.276350+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:11.903925+0000 mgr.y (mgr.25138) 3 : cluster [DBG] pgmap v5: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.322426+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.592998+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.593356+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.593610+0000 mon.a (mon.0) 71 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.593892+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.594148+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.594387+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.594628+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.594877+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.595132+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:12.982419+0000 mon.a (mon.0) 78 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.260719+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.261124+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.261461+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.261799+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.262121+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.262441+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.262714+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.263006+0000 mon.a (mon.0) 86 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.263302+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "osd_memory_target_autotune"}]: dispatch 2024-02-10T09:46:13.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.267405+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:13.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.272643+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:13.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.274740+0000 mon.a (mon.0) 90 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:13.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.275756+0000 mon.a (mon.0) 91 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T09:46:13.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:13.276350+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T09:46:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:46:14.651 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:13.276785+0000 mgr.y (mgr.25138) 4 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2024-02-10T09:46:14.652 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:14] "GET /metrics HTTP/1.1" 200 153059 "" "Prometheus/2.33.4" 2024-02-10T09:46:14.652 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:14 smithi153 bash[39117]: cephadm 2024-02-10T09:46:13.276785+0000 mgr.y (mgr.25138) 4 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2024-02-10T09:46:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:15 smithi153 bash[39117]: cluster 2024-02-10T09:46:13.904545+0000 mgr.y (mgr.25138) 5 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:15 smithi153 bash[39117]: audit 2024-02-10T09:46:14.673333+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:15 smithi153 bash[39117]: audit 2024-02-10T09:46:14.673995+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:15 smithi153 bash[39117]: audit 2024-02-10T09:46:14.674505+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:15 smithi153 bash[39117]: audit 2024-02-10T09:46:14.674992+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:15 smithi153 bash[39117]: audit 2024-02-10T09:46:14.675840+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:46:15.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:13.904545+0000 mgr.y (mgr.25138) 5 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:15.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:14.673333+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:15.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:14.673995+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:15.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:14.674505+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:15.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:14.674992+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:15.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:14.675840+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:46:16.225 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:16 smithi153 systemd[1]: Stopping Ceph mon.c for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:46:16.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:16 smithi153 bash[39117]: debug 2024-02-10T09:46:16.223+0000 7f9039de4700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:46:16.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:16 smithi153 bash[39117]: debug 2024-02-10T09:46:16.223+0000 7f9039de4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T09:46:16.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:16 smithi153 bash[73250]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.c 2024-02-10T09:46:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.c.service: Succeeded. 2024-02-10T09:46:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 systemd[1]: Stopped Ceph mon.c for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:46:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 systemd[1]: Starting Ceph mon.c for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.958+0000 7febb2545880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.958+0000 7febb2545880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.958+0000 7febb2545880 0 pidfile_write: ignore empty --pid-file 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.962+0000 7febb2545880 0 load: jerasure load: lrc load: isa 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T09:46:18.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: DB SUMMARY 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: CURRENT file: CURRENT 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 3023 Bytes 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 2, files: 000069.sst 000070.sst 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000067.log size: 8838485 ; 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:46:18.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.create_if_missing: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.env: 0x55ea61c86080 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.fs: Posix File System 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.info_log: 0x55ea63732260 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:46:18.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.db_log_dir: 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:46:18.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.write_buffer_manager: 0x55ea64424390 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.unordered_write: 0 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:46:18.103 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.row_cache: None 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.wal_filter: None 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:46:18.105 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:46:18.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Compression algorithms supported: 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kZSTD supported: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.merge_operator: 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ea636d7d28) 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cache_index_and_filter_blocks: 1 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: pin_top_level_index_and_filter: 1 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: index_type: 0 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: data_block_index_type: 0 2024-02-10T09:46:18.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: index_shortening: 1 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: hash_index_allow_collision: 1 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: checksum: 1 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: no_block_cache: 0 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_cache: 0x55ea6370cf10 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_cache_name: BinnedLRUCache 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_cache_options: 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: capacity : 536870912 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: num_shard_bits : 4 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: strict_capacity_limit : 0 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: high_pri_pool_ratio: 0.000 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_cache_compressed: (nil) 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: persistent_cache: (nil) 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_size: 4096 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_size_deviation: 10 2024-02-10T09:46:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_restart_interval: 16 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: index_block_restart_interval: 1 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: metadata_block_size: 4096 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: partition_filters: 0 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: use_delta_encoding: 1 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: whole_key_filtering: 1 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: verify_compression: 0 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: read_amp_bytes_per_bit: 0 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: format_version: 2 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: enable_index_compression: 1 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: block_align: 0 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:46:18.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.num_levels: 7 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:46:18.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:46:18.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:46:18.113 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:46:18.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.ttl: 2592000 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.963+0000 7febb2545880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.964+0000 7febb2545880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 72, last_sequence is 25448, log_number is 67,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.964+0000 7febb2545880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 67 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.964+0000 7febb2545880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558377965497, "job": 1, "event": "recovery_started", "log_files": [67]} 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.964+0000 7febb2545880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #67 mode 2 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.974+0000 7febb2545880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.983+0000 7febb2545880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558377984016, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 72, "file_size": 5868869, "table_properties": {"data_size": 5862161, "index_size": 5052, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 837, "raw_key_size": 7606, "raw_average_key_size": 24, "raw_value_size": 5853364, "raw_average_value_size": 19191, "num_data_blocks": 157, "num_entries": 305, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707558377, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.983+0000 7febb2545880 4 rocksdb: [version_set.cc:3826] Creating manifest 73 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.985+0000 7febb2545880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707558377986465, "job": 1, "event": "recovery_finished"} 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.987+0000 7febb2545880 4 rocksdb: DB pointer 0x55ea637d5800 2024-02-10T09:46:18.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.987+0000 7feb99336700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.987+0000 7feb99336700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** DB Stats ** 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** Compaction Stats [default] ** 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:46:18.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: L0 1/0 5.60 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: L6 2/0 118.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Sum 3/0 123.99 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** Compaction Stats [default] ** 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Flush(GB): cumulative 0.005, interval 0.005 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:46:18.117 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Cumulative compaction: 0.01 GB write, 239.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Interval compaction: 0.01 GB write, 239.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** Compaction Stats [default] ** 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: L0 1/0 5.60 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: L6 2/0 118.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Sum 3/0 123.99 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** Compaction Stats [default] ** 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:46:18.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 611.2 0.01 0.00 1 0.009 0 0 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Flush(GB): cumulative 0.005, interval 0.000 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Cumulative compaction: 0.01 GB write, 239.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.987+0000 7febb2545880 0 starting mon.c rank 1 at public addrs [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] at bind addrs [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.988+0000 7febb2545880 1 mon.c@-1(???) e3 preinit fsid 09b43244-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:46:18.119 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 0 mon.c@-1(???).mds e1 new map 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 0 mon.c@-1(???).mds e1 print_map 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: e1 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: enable_multiple, ever_enabled_multiple: 0,0 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: legacy client fscid: -1 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: No filesystems configured 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 0 mon.c@-1(???).osd e73 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 0 mon.c@-1(???).osd e73 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 0 mon.c@-1(???).osd e73 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 0 mon.c@-1(???).osd e73 crush map has features 288514051259236352, adjusting msgr requires 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:13.904545+0000 mgr.y (mgr.25138) 5 : cluster [DBG] pgmap v6: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:18.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:14.673333+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:14.673995+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:14.674505+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:14.674992+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:14.675840+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:46:17.989+0000 7febb2545880 1 mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:18 smithi153 bash[73436]: e4f1164dd253457e9edf870882de4f1c000cc12b189dbf1e9245051a3c36ad5e 2024-02-10T09:46:18.121 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:18 smithi153 systemd[1]: Started Ceph mon.c for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.371502+0000 mon.a (mon.0) 104 : cluster [INF] overall HEALTH_OK 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:19.372036+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.372160+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.372249+0000 mon.b (mon.2) 6 : cluster [INF] mon.b calling monitor election 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.372359+0000 mon.a (mon.0) 106 : cluster [INF] mon.a calling monitor election 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:19.373374+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:19.373496+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.386331+0000 mon.a (mon.0) 109 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:46:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.390591+0000 mon.a (mon.0) 110 : cluster [DBG] monmap e4: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} removed_ranks: {} 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.391054+0000 mon.a (mon.0) 111 : cluster [DBG] fsmap 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.391102+0000 mon.a (mon.0) 112 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.391476+0000 mon.a (mon.0) 113 : cluster [DBG] mgrmap e45: y(active, since 17s), standbys: x 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.394675+0000 mon.a (mon.0) 114 : cluster [INF] overall HEALTH_OK 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:19.397632+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:19.398675+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:19.400331+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.371502+0000 mon.a (mon.0) 104 : cluster [INF] overall HEALTH_OK 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:19.372036+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.372160+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T09:46:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.372249+0000 mon.b (mon.2) 6 : cluster [INF] mon.b calling monitor election 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.372359+0000 mon.a (mon.0) 106 : cluster [INF] mon.a calling monitor election 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:19.373374+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:19.373496+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.386331+0000 mon.a (mon.0) 109 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.390591+0000 mon.a (mon.0) 110 : cluster [DBG] monmap e4: 3 mons at {a=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],b=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0],c=[v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0]} removed_ranks: {} 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.391054+0000 mon.a (mon.0) 111 : cluster [DBG] fsmap 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.391102+0000 mon.a (mon.0) 112 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.391476+0000 mon.a (mon.0) 113 : cluster [DBG] mgrmap e45: y(active, since 17s), standbys: x 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.394675+0000 mon.a (mon.0) 114 : cluster [INF] overall HEALTH_OK 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:19.397632+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:19.398675+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:20.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:19.400331+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:46:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:19.906163+0000 mgr.y (mgr.25138) 10 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:21.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:19.906163+0000 mgr.y (mgr.25138) 10 : cluster [DBG] pgmap v9: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:21.894 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:46:22.555 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:46:22.555 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (14m) 10s ago 15m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:46:22.555 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (2m) 9s ago 17m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:46:22.555 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (2m) 9s ago 27m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:46:22.555 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (10m) 10s ago 29m 411M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (27s) 10s ago 29m 34.6M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (72s) 9s ago 28m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 starting - - - 2048M 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (2m) 10s ago 20m 14.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (2m) 9s ago 20m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (27m) 10s ago 27m 44.0M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (26m) 10s ago 27m 44.5M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (26m) 10s ago 26m 43.5M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (26m) 10s ago 26m 46.1M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (26m) 9s ago 26m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (25m) 9s ago 25m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (25m) 9s ago 25m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:46:22.556 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (25m) 9s ago 25m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:46:22.557 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (14m) 9s ago 14m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:46:22.557 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (17m) 10s ago 17m 36.9M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:46:22.557 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (17m) 9s ago 17m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:46:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 10, 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 5 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:46:22.960 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi153", 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "1/2 daemons upgraded", 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:46:23.281 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:46:23.404 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:46:22.994+0000 7fc4ea3a1700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-10T09:46:23.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:21.882265+0000 mgr.y (mgr.25138) 11 : audit [DBG] from='client.54126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:21.906559+0000 mgr.y (mgr.25138) 12 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.209354+0000 mgr.y (mgr.25138) 13 : audit [DBG] from='client.54132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.541510+0000 mgr.y (mgr.25138) 14 : audit [DBG] from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.543475+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.543927+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.544303+0000 mon.a (mon.0) 120 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.544674+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.545017+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.545345+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.546966+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.547316+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.547614+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.547883+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.548141+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:22.958812+0000 mon.c (mon.1) 3 : audit [DBG] from='client.54144 172.21.15.153:0/2901194100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.379491+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.384297+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.386863+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.387445+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.387956+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.388735+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:21.882265+0000 mgr.y (mgr.25138) 11 : audit [DBG] from='client.54126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:21.906559+0000 mgr.y (mgr.25138) 12 : cluster [DBG] pgmap v10: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.209354+0000 mgr.y (mgr.25138) 13 : audit [DBG] from='client.54132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:23.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.541510+0000 mgr.y (mgr.25138) 14 : audit [DBG] from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.543475+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.543927+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.544303+0000 mon.a (mon.0) 120 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.544674+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.545017+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.545345+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.546966+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.547316+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.547614+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.547883+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.548141+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:23.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:22.958812+0000 mon.c (mon.1) 3 : audit [DBG] from='client.54144 172.21.15.153:0/2901194100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:23.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.379491+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:23.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.384297+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:23.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.386863+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:23.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.387445+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:23.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.387956+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:23.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.388735+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "container_image"}]: dispatch 2024-02-10T09:46:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:46:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:24] "GET /metrics HTTP/1.1" 200 150099 "" "Prometheus/2.33.4" 2024-02-10T09:46:24.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:23.280441+0000 mgr.y (mgr.25138) 15 : audit [DBG] from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:24.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:23.386647+0000 mgr.y (mgr.25138) 16 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2024-02-10T09:46:24.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:23.388391+0000 mgr.y (mgr.25138) 17 : cephadm [INF] Reconfiguring daemon mon.a on smithi153 2024-02-10T09:46:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:23.280441+0000 mgr.y (mgr.25138) 15 : audit [DBG] from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:23.386647+0000 mgr.y (mgr.25138) 16 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2024-02-10T09:46:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:23.388391+0000 mgr.y (mgr.25138) 17 : cephadm [INF] Reconfiguring daemon mon.a on smithi153 2024-02-10T09:46:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:23.907362+0000 mgr.y (mgr.25138) 18 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:24.771183+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:24.772177+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:46:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:24.772854+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:46:25.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:24.773478+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:24.774319+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.y","key": "container_image"}]: dispatch 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:23.907362+0000 mgr.y (mgr.25138) 18 : cluster [DBG] pgmap v11: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:24.771183+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:24.772177+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:24.772854+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:24.773478+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:25.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:24.774319+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.y","key": "container_image"}]: dispatch 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:24.771930+0000 mgr.y (mgr.25138) 19 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:24.773937+0000 mgr.y (mgr.25138) 20 : cephadm [INF] Reconfiguring daemon mgr.y on smithi153 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:26.045655+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:26.046554+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:26.047090+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:26.047549+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:26.048326+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:24.771930+0000 mgr.y (mgr.25138) 19 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:24.773937+0000 mgr.y (mgr.25138) 20 : cephadm [INF] Reconfiguring daemon mgr.y on smithi153 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:26.045655+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:26.046554+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:26.047090+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:26.047549+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:26.048326+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "container_image"}]: dispatch 2024-02-10T09:46:27.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:25.907956+0000 mgr.y (mgr.25138) 21 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:27.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:26.046338+0000 mgr.y (mgr.25138) 22 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T09:46:27.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:26.047987+0000 mgr.y (mgr.25138) 23 : cephadm [INF] Reconfiguring daemon mon.c on smithi153 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:27.371377+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:27.372325+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:27.372863+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:27.374307+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "container_image"}]: dispatch 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:25.907956+0000 mgr.y (mgr.25138) 21 : cluster [DBG] pgmap v12: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:26.046338+0000 mgr.y (mgr.25138) 22 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:26.047987+0000 mgr.y (mgr.25138) 23 : cephadm [INF] Reconfiguring daemon mon.c on smithi153 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:27.371377+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:27.372325+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:27.372863+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:27.374307+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "container_image"}]: dispatch 2024-02-10T09:46:28.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:27.372095+0000 mgr.y (mgr.25138) 24 : cephadm [INF] Reconfiguring osd.0 (monmap changed)... 2024-02-10T09:46:28.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:27.373942+0000 mgr.y (mgr.25138) 25 : cephadm [INF] Reconfiguring daemon osd.0 on smithi153 2024-02-10T09:46:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:27.372095+0000 mgr.y (mgr.25138) 24 : cephadm [INF] Reconfiguring osd.0 (monmap changed)... 2024-02-10T09:46:28.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:27.373942+0000 mgr.y (mgr.25138) 25 : cephadm [INF] Reconfiguring daemon osd.0 on smithi153 2024-02-10T09:46:29.474 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:27.908304+0000 mgr.y (mgr.25138) 26 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:29.474 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:28.773596+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:29.474 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:28.774552+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:46:29.474 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:28.775136+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:29.475 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:28.776455+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "container_image"}]: dispatch 2024-02-10T09:46:29.475 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:27.908304+0000 mgr.y (mgr.25138) 26 : cluster [DBG] pgmap v13: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:29.475 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:28.773596+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:29.475 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:28.774552+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:46:29.475 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:28.775136+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:29.475 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:28.776455+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "container_image"}]: dispatch 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:28.774319+0000 mgr.y (mgr.25138) 27 : cephadm [INF] Reconfiguring osd.1 (monmap changed)... 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:28.776094+0000 mgr.y (mgr.25138) 28 : cephadm [INF] Reconfiguring daemon osd.1 on smithi153 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:30.164817+0000 mon.a (mon.0) 153 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:30.165709+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:30.166207+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:30.167515+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "container_image"}]: dispatch 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:28.774319+0000 mgr.y (mgr.25138) 27 : cephadm [INF] Reconfiguring osd.1 (monmap changed)... 2024-02-10T09:46:30.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:28.776094+0000 mgr.y (mgr.25138) 28 : cephadm [INF] Reconfiguring daemon osd.1 on smithi153 2024-02-10T09:46:30.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:30.164817+0000 mon.a (mon.0) 153 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:30.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:30.165709+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:46:30.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:30.166207+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:30.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:30.167515+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "container_image"}]: dispatch 2024-02-10T09:46:31.506 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:29.908978+0000 mgr.y (mgr.25138) 29 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:31.506 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:30.165481+0000 mgr.y (mgr.25138) 30 : cephadm [INF] Reconfiguring osd.2 (monmap changed)... 2024-02-10T09:46:31.506 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:30.167143+0000 mgr.y (mgr.25138) 31 : cephadm [INF] Reconfiguring daemon osd.2 on smithi153 2024-02-10T09:46:31.506 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:29.908978+0000 mgr.y (mgr.25138) 29 : cluster [DBG] pgmap v14: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:31.506 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:30.165481+0000 mgr.y (mgr.25138) 30 : cephadm [INF] Reconfiguring osd.2 (monmap changed)... 2024-02-10T09:46:31.507 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:30.167143+0000 mgr.y (mgr.25138) 31 : cephadm [INF] Reconfiguring daemon osd.2 on smithi153 2024-02-10T09:46:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:31.572027+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:31.572695+0000 mgr.y (mgr.25138) 32 : cephadm [INF] Reconfiguring osd.3 (monmap changed)... 2024-02-10T09:46:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:31.572906+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:46:32.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:31.573418+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:31.574375+0000 mgr.y (mgr.25138) 33 : cephadm [INF] Reconfiguring daemon osd.3 on smithi153 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:31.574706+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "container_image"}]: dispatch 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:31.909353+0000 mgr.y (mgr.25138) 34 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:31.572027+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:31.572695+0000 mgr.y (mgr.25138) 32 : cephadm [INF] Reconfiguring osd.3 (monmap changed)... 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:31.572906+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:31.573418+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:31.574375+0000 mgr.y (mgr.25138) 33 : cephadm [INF] Reconfiguring daemon osd.3 on smithi153 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:31.574706+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "container_image"}]: dispatch 2024-02-10T09:46:32.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:31.909353+0000 mgr.y (mgr.25138) 34 : cluster [DBG] pgmap v15: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.889093+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:32.889876+0000 mgr.y (mgr.25138) 35 : cephadm [INF] Reconfiguring rgw.r.z.smithi153.zyveqt (monmap changed)... 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.890169+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.890828+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.896025+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:32.896511+0000 mgr.y (mgr.25138) 36 : cephadm [WRN] Unable to update caps for client.rgw.r.z.smithi153.zyveqt 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.902204+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.902787+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.rgw.r.z.smithi153.zyveqt"}]: dispatch 2024-02-10T09:46:34.134 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.903263+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:32.903715+0000 mgr.y (mgr.25138) 37 : cephadm [INF] Reconfiguring daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:32.904021+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi153.zyveqt","key": "container_image"}]: dispatch 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.889093+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:32.889876+0000 mgr.y (mgr.25138) 35 : cephadm [INF] Reconfiguring rgw.r.z.smithi153.zyveqt (monmap changed)... 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.890169+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.890828+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.896025+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:32.896511+0000 mgr.y (mgr.25138) 36 : cephadm [WRN] Unable to update caps for client.rgw.r.z.smithi153.zyveqt 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.902204+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.902787+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.rgw.r.z.smithi153.zyveqt"}]: dispatch 2024-02-10T09:46:34.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.903263+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:34.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:32.903715+0000 mgr.y (mgr.25138) 37 : cephadm [INF] Reconfiguring daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:46:34.136 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:32.904021+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi153.zyveqt","key": "container_image"}]: dispatch 2024-02-10T09:46:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:34] "GET /metrics HTTP/1.1" 200 150099 "" "Prometheus/2.33.4" 2024-02-10T09:46:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:33.910096+0000 mgr.y (mgr.25138) 38 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:34.174801+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:34.175745+0000 mgr.y (mgr.25138) 39 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:34.175954+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:34.176477+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:34.176970+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:34.177412+0000 mgr.y (mgr.25138) 40 : cephadm [INF] Reconfiguring daemon mon.b on smithi204 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:34.177705+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:33.910096+0000 mgr.y (mgr.25138) 38 : cluster [DBG] pgmap v16: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:34.174801+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:34.175745+0000 mgr.y (mgr.25138) 39 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:34.175954+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T09:46:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:34.176477+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T09:46:35.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:34.176970+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:35.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:34.177412+0000 mgr.y (mgr.25138) 40 : cephadm [INF] Reconfiguring daemon mon.b on smithi204 2024-02-10T09:46:35.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:34.177705+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "container_image"}]: dispatch 2024-02-10T09:46:36.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:35.670895+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:36.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:35.671769+0000 mgr.y (mgr.25138) 41 : cephadm [INF] Reconfiguring mgr.x (monmap changed)... 2024-02-10T09:46:36.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:35.672085+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:46:36.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:35.672980+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:35.673662+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:35.674288+0000 mgr.y (mgr.25138) 42 : cephadm [INF] Reconfiguring daemon mgr.x on smithi204 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:35.674754+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.x","key": "container_image"}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:35.910594+0000 mgr.y (mgr.25138) 43 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:35.670895+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:35.671769+0000 mgr.y (mgr.25138) 41 : cephadm [INF] Reconfiguring mgr.x (monmap changed)... 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:35.672085+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:35.672980+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:35.673662+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:35.674288+0000 mgr.y (mgr.25138) 42 : cephadm [INF] Reconfiguring daemon mgr.x on smithi204 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:35.674754+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mgr.x","key": "container_image"}]: dispatch 2024-02-10T09:46:36.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:35.910594+0000 mgr.y (mgr.25138) 43 : cluster [DBG] pgmap v17: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:38.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:37.058080+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:38.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:37.058954+0000 mgr.y (mgr.25138) 44 : cephadm [INF] Reconfiguring osd.4 (monmap changed)... 2024-02-10T09:46:38.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:37.059276+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:37.060099+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:37.061438+0000 mgr.y (mgr.25138) 45 : cephadm [INF] Reconfiguring daemon osd.4 on smithi204 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:37.061922+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "container_image"}]: dispatch 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:37.058080+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:37.058954+0000 mgr.y (mgr.25138) 44 : cephadm [INF] Reconfiguring osd.4 (monmap changed)... 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:37.059276+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:37.060099+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:37.061438+0000 mgr.y (mgr.25138) 45 : cephadm [INF] Reconfiguring daemon osd.4 on smithi204 2024-02-10T09:46:38.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:37.061922+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "container_image"}]: dispatch 2024-02-10T09:46:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:37.911057+0000 mgr.y (mgr.25138) 46 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:38.485552+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:38.486441+0000 mgr.y (mgr.25138) 47 : cephadm [INF] Reconfiguring osd.5 (monmap changed)... 2024-02-10T09:46:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:38.486806+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:46:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:38.487495+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:38.488721+0000 mgr.y (mgr.25138) 48 : cephadm [INF] Reconfiguring daemon osd.5 on smithi204 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:38.489193+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "container_image"}]: dispatch 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:37.911057+0000 mgr.y (mgr.25138) 46 : cluster [DBG] pgmap v18: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:38.485552+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:38.486441+0000 mgr.y (mgr.25138) 47 : cephadm [INF] Reconfiguring osd.5 (monmap changed)... 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:38.486806+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:38.487495+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:38.488721+0000 mgr.y (mgr.25138) 48 : cephadm [INF] Reconfiguring daemon osd.5 on smithi204 2024-02-10T09:46:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:38.489193+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "container_image"}]: dispatch 2024-02-10T09:46:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:39.911809+0000 mgr.y (mgr.25138) 49 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:39.981855+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:39.982861+0000 mgr.y (mgr.25138) 50 : cephadm [INF] Reconfiguring osd.6 (monmap changed)... 2024-02-10T09:46:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:39.983274+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:46:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:39.984072+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:39.985570+0000 mgr.y (mgr.25138) 51 : cephadm [INF] Reconfiguring daemon osd.6 on smithi204 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:39.986067+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "container_image"}]: dispatch 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:39.911809+0000 mgr.y (mgr.25138) 49 : cluster [DBG] pgmap v19: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:39.981855+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:39.982861+0000 mgr.y (mgr.25138) 50 : cephadm [INF] Reconfiguring osd.6 (monmap changed)... 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:39.983274+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:39.984072+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:39.985570+0000 mgr.y (mgr.25138) 51 : cephadm [INF] Reconfiguring daemon osd.6 on smithi204 2024-02-10T09:46:41.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:39.986067+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "container_image"}]: dispatch 2024-02-10T09:46:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:41.374714+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:41.375577+0000 mgr.y (mgr.25138) 52 : cephadm [INF] Reconfiguring osd.7 (monmap changed)... 2024-02-10T09:46:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:41.375880+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:46:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:41.376755+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:41.377900+0000 mgr.y (mgr.25138) 53 : cephadm [INF] Reconfiguring daemon osd.7 on smithi204 2024-02-10T09:46:42.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:41.378286+0000 mon.a (mon.0) 194 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "container_image"}]: dispatch 2024-02-10T09:46:42.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:41.374714+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:42.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:41.375577+0000 mgr.y (mgr.25138) 52 : cephadm [INF] Reconfiguring osd.7 (monmap changed)... 2024-02-10T09:46:42.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:41.375880+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:46:42.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:41.376755+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:42.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:41.377900+0000 mgr.y (mgr.25138) 53 : cephadm [INF] Reconfiguring daemon osd.7 on smithi204 2024-02-10T09:46:42.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:41.378286+0000 mon.a (mon.0) 194 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "container_image"}]: dispatch 2024-02-10T09:46:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:41.912311+0000 mgr.y (mgr.25138) 54 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.671171+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.672585+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.673484+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.679161+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.685722+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.686505+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.rgw.r.z.smithi204.kmhqqg"}]: dispatch 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.687184+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:42.688296+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi204.kmhqqg","key": "container_image"}]: dispatch 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:41.912311+0000 mgr.y (mgr.25138) 54 : cluster [DBG] pgmap v20: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.671171+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.672585+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.673484+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.679161+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "auth caps", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.685722+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.686505+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.rgw.r.z.smithi204.kmhqqg"}]: dispatch 2024-02-10T09:46:43.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.687184+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:46:43.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:42.688296+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi204.kmhqqg","key": "container_image"}]: dispatch 2024-02-10T09:46:43.959 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:46:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:44] "GET /metrics HTTP/1.1" 200 150111 "" "Prometheus/2.33.4" 2024-02-10T09:46:44.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:42.672204+0000 mgr.y (mgr.25138) 55 : cephadm [INF] Reconfiguring rgw.r.z.smithi204.kmhqqg (monmap changed)... 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:42.679790+0000 mgr.y (mgr.25138) 56 : cephadm [WRN] Unable to update caps for client.rgw.r.z.smithi204.kmhqqg 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:42.687849+0000 mgr.y (mgr.25138) 57 : cephadm [INF] Reconfiguring daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.001347+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.003396+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.006004+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.007457+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.012186+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.013007+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.017214+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.017998+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.021669+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.022769+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.027626+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.029139+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.033612+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:42.672204+0000 mgr.y (mgr.25138) 55 : cephadm [INF] Reconfiguring rgw.r.z.smithi204.kmhqqg (monmap changed)... 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:42.679790+0000 mgr.y (mgr.25138) 56 : cephadm [WRN] Unable to update caps for client.rgw.r.z.smithi204.kmhqqg 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:42.687849+0000 mgr.y (mgr.25138) 57 : cephadm [INF] Reconfiguring daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.001347+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.003396+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.006004+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.007457+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.012186+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.013007+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.017214+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.017998+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.021669+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.022769+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.027626+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.029139+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.033612+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.034942+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.041451+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.042452+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.044012+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.044606+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.045440+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.051971+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.052997+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.059645+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.060900+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.062399+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.062942+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.063901+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.065166+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.065746+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.066711+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.034942+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.041451+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.042452+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.044012+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.044606+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.045440+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.051971+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.052997+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.059645+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.060900+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.062399+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.062942+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.063901+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.065166+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.065746+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.066711+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.068042+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.068577+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.069618+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.070990+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.071527+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.072694+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.078526+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.079255+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.084885+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.085568+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.091012+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.091656+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.092234+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.097847+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.098558+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:46:44.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.099175+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.104945+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.105534+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.106140+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.111584+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.112222+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.117695+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.118331+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.118951+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.119510+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.120100+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.120663+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:46:44.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.123947+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.068042+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.068577+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.069618+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.070990+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.071527+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.072694+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.078526+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.079255+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.084885+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.085568+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.091012+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.091656+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:46:44.698 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.092234+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.097847+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.124831+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.127434+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.127838+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.132430+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.136894+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:44.699 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:44.141445+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.098558+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.099175+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.104945+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.105534+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.106140+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.111584+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.112222+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.117695+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:46:44.700 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.118331+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.118951+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.119510+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.120100+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.120663+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.123947+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.124831+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.127434+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.127838+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.132430+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.136894+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:46:44.701 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:44.141445+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:43.912998+0000 mgr.y (mgr.25138) 58 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.008102+0000 mgr.y (mgr.25138) 59 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:46:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.023468+0000 mgr.y (mgr.25138) 60 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:46:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.029738+0000 mgr.y (mgr.25138) 61 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:46:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.035581+0000 mgr.y (mgr.25138) 62 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:46:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.043127+0000 mgr.y (mgr.25138) 63 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.046001+0000 mgr.y (mgr.25138) 64 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.053655+0000 mgr.y (mgr.25138) 65 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.061527+0000 mgr.y (mgr.25138) 66 : cephadm [INF] Upgrade: Setting container_image for all node-exporter 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.064425+0000 mgr.y (mgr.25138) 67 : cephadm [INF] Upgrade: Setting container_image for all prometheus 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.067252+0000 mgr.y (mgr.25138) 68 : cephadm [INF] Upgrade: Setting container_image for all alertmanager 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.070214+0000 mgr.y (mgr.25138) 69 : cephadm [INF] Upgrade: Setting container_image for all grafana 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.072011+0000 mgr.y (mgr.25138) 70 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:46:44.124452+0000 mgr.y (mgr.25138) 71 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:43.912998+0000 mgr.y (mgr.25138) 58 : cluster [DBG] pgmap v21: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.008102+0000 mgr.y (mgr.25138) 59 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.023468+0000 mgr.y (mgr.25138) 60 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:46:45.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.029738+0000 mgr.y (mgr.25138) 61 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.035581+0000 mgr.y (mgr.25138) 62 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.043127+0000 mgr.y (mgr.25138) 63 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.046001+0000 mgr.y (mgr.25138) 64 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.053655+0000 mgr.y (mgr.25138) 65 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.061527+0000 mgr.y (mgr.25138) 66 : cephadm [INF] Upgrade: Setting container_image for all node-exporter 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.064425+0000 mgr.y (mgr.25138) 67 : cephadm [INF] Upgrade: Setting container_image for all prometheus 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.067252+0000 mgr.y (mgr.25138) 68 : cephadm [INF] Upgrade: Setting container_image for all alertmanager 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.070214+0000 mgr.y (mgr.25138) 69 : cephadm [INF] Upgrade: Setting container_image for all grafana 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.072011+0000 mgr.y (mgr.25138) 70 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:46:45.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:46:44.124452+0000 mgr.y (mgr.25138) 71 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:46:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:45.913480+0000 mgr.y (mgr.25138) 72 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:45.913480+0000 mgr.y (mgr.25138) 72 : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:49.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:47.914047+0000 mgr.y (mgr.25138) 73 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:49.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:48.088216+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:49.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:47.914047+0000 mgr.y (mgr.25138) 73 : cluster [DBG] pgmap v23: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:49.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:48.088216+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:46:51.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:49.914770+0000 mgr.y (mgr.25138) 74 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:51.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:49.914770+0000 mgr.y (mgr.25138) 74 : cluster [DBG] pgmap v24: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:51.915284+0000 mgr.y (mgr.25138) 75 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:53.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:51.915284+0000 mgr.y (mgr.25138) 75 : cluster [DBG] pgmap v25: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:46:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:46:54.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:46:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:46:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:46:54] "GET /metrics HTTP/1.1" 200 150121 "" "Prometheus/2.33.4" 2024-02-10T09:46:55.328 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:53.612118+0000 mgr.y (mgr.25138) 76 : audit [DBG] from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:55.328 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:53.916028+0000 mgr.y (mgr.25138) 77 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:55.328 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:53.612118+0000 mgr.y (mgr.25138) 76 : audit [DBG] from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:55.329 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:53.916028+0000 mgr.y (mgr.25138) 77 : cluster [DBG] pgmap v26: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:56.255 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (14m) 32s ago 15m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (2m) 43s ago 17m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (3m) 43s ago 28m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (11m) 32s ago 30m 416M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (61s) 32s ago 30m 39.2M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (106s) 43s ago 28m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (38s) 32s ago 28m 25.7M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (3m) 32s ago 20m 14.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (2m) 43s ago 20m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:46:56.256 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (27m) 32s ago 28m 44.4M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (27m) 32s ago 27m 44.8M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (27m) 32s ago 27m 43.5M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (26m) 32s ago 27m 46.5M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (26m) 43s ago 26m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (26m) 43s ago 26m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (26m) 43s ago 26m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (25m) 43s ago 25m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (15m) 43s ago 15m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (17m) 32s ago 17m 37.0M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:46:56.257 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (17m) 43s ago 17m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:46:56.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-02-10T09:46:57.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:55.916428+0000 mgr.y (mgr.25138) 78 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.244287+0000 mgr.y (mgr.25138) 79 : audit [DBG] from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.246000+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.246338+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.246606+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.246898+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.247163+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.247421+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.248937+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.249205+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.249470+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.249726+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:56.249997+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:55.916428+0000 mgr.y (mgr.25138) 78 : cluster [DBG] pgmap v27: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.244287+0000 mgr.y (mgr.25138) 79 : audit [DBG] from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.246000+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.246338+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.246606+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.246898+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.247163+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.247421+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.248937+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.249205+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.249470+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.249726+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:57.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:56.249997+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:46:59.018 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:46:59.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:57.916809+0000 mgr.y (mgr.25138) 80 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:59.356 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:46:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:46:59.006790+0000 mon.c (mon.1) 4 : audit [DBG] from='client.54168 172.21.15.153:0/3037416667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:59.356 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:57.916809+0000 mgr.y (mgr.25138) 80 : cluster [DBG] pgmap v28: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:46:59.356 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:46:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:46:59.006790+0000 mon.c (mon.1) 4 : audit [DBG] from='client.54168 172.21.15.153:0/3037416667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:46:59.664 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-02-10T09:47:01.179 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:46:59.917424+0000 mgr.y (mgr.25138) 81 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:01.180 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:46:59.917424+0000 mgr.y (mgr.25138) 81 : cluster [DBG] pgmap v29: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)" 2024-02-10T09:47:02.327 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:01.659974+0000 mon.c (mon.1) 5 : audit [DBG] from='client.54174 172.21.15.153:0/3861482751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:02.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:01.659974+0000 mon.c (mon.1) 5 : audit [DBG] from='client.54174 172.21.15.153:0/3861482751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:02.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-02-10T09:47:03.426 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:01.917739+0000 mgr.y (mgr.25138) 82 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:03.426 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:01.917739+0000 mgr.y (mgr.25138) 82 : cluster [DBG] pgmap v30: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:47:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:47:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:04] "GET /metrics HTTP/1.1" 200 150121 "" "Prometheus/2.33.4" 2024-02-10T09:47:05.379 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:03.918329+0000 mgr.y (mgr.25138) 83 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:05.379 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:04.088116+0000 mgr.y (mgr.25138) 84 : audit [DBG] from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:05.380 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:03.918329+0000 mgr.y (mgr.25138) 83 : cluster [DBG] pgmap v31: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:05.380 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:04.088116+0000 mgr.y (mgr.25138) 84 : audit [DBG] from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:47:06.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-02-10T09:47:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:05.918740+0000 mgr.y (mgr.25138) 85 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:05.918740+0000 mgr.y (mgr.25138) 85 : cluster [DBG] pgmap v32: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:07.922606+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-02-10T09:47:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:07.922946+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:47:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:07.923230+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:47:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:07.923530+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:47:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:07.922606+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-02-10T09:47:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:07.922946+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:47:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:07.923230+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:47:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:07.923530+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T09:47:09.408 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:07.919357+0000 mgr.y (mgr.25138) 86 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:09.408 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:08.485032+0000 mgr.y (mgr.25138) 87 : audit [DBG] from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:09.408 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:07.919357+0000 mgr.y (mgr.25138) 86 : cluster [DBG] pgmap v33: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:09.408 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:08.485032+0000 mgr.y (mgr.25138) 87 : audit [DBG] from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:10.315 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:47:10.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:47:11.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:09.919982+0000 mgr.y (mgr.25138) 88 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:11.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:10.310789+0000 mgr.y (mgr.25138) 89 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:47:11.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:10.314350+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:11.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:10.316933+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:11.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:10.322231+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:11.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:10.325872+0000 mgr.y (mgr.25138) 90 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:47:11.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:09.919982+0000 mgr.y (mgr.25138) 88 : cluster [DBG] pgmap v34: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:11.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:10.310789+0000 mgr.y (mgr.25138) 89 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:47:11.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:10.314350+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:11.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:10.316933+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:11.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:10.322231+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:11.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:10.325872+0000 mgr.y (mgr.25138) 90 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:47:13.142 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:47:13.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:11.920413+0000 mgr.y (mgr.25138) 91 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:13.534 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:11.920413+0000 mgr.y (mgr.25138) 91 : cluster [DBG] pgmap v35: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:13.781 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (15m) 50s ago 16m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (3m) 60s ago 18m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (3m) 60s ago 28m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (11m) 50s ago 30m 416M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (79s) 50s ago 30m 39.2M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (2m) 60s ago 28m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (56s) 50s ago 29m 25.7M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (3m) 50s ago 20m 14.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (3m) 60s ago 20m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (27m) 50s ago 28m 44.4M 4096M 15.2.0 204a01f9b0b6 90f82f6e81ca 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (27m) 50s ago 28m 44.8M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (27m) 50s ago 27m 43.5M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:47:13.782 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (27m) 50s ago 27m 46.5M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (27m) 60s ago 27m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (26m) 60s ago 26m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (26m) 60s ago 26m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (26m) 60s ago 26m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (15m) 60s ago 15m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (18m) 50s ago 18m 37.0M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:47:13.783 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (18m) 60s ago 18m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:47:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:47:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:47:14.183 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:47:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:47:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 10, 2024-02-10T09:47:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 5 2024-02-10T09:47:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:47:14.185 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/8 daemons upgraded", 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:47:14.516 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:47:14.517 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:14] "GET /metrics HTTP/1.1" 200 150066 "" "Prometheus/2.33.4" 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.129344+0000 mgr.y (mgr.25138) 92 : audit [DBG] from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.457765+0000 mgr.y (mgr.25138) 93 : audit [DBG] from='client.64150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.771620+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.771984+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.772282+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.772580+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.772878+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.773160+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.774740+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.775043+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.775336+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.775617+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.775904+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.869925+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.129344+0000 mgr.y (mgr.25138) 92 : audit [DBG] from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.457765+0000 mgr.y (mgr.25138) 93 : audit [DBG] from='client.64150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:14.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.771620+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.771984+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.772282+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.772580+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.772878+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.773160+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.774740+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.775043+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.775336+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.775617+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.775904+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.869925+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.871598+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.878981+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.885121+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.891768+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.892872+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.898919+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.899946+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.907815+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.909078+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:14.183404+0000 mon.a (mon.0) 306 : audit [DBG] from='client.54210 172.21.15.153:0/2761566060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.871598+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.878981+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.885121+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.891768+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.892872+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.898919+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.899946+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.907815+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.909078+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T09:47:14.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:14.183404+0000 mon.a (mon.0) 306 : audit [DBG] from='client.54210 172.21.15.153:0/2761566060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.769797+0000 mgr.y (mgr.25138) 94 : audit [DBG] from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:13.870248+0000 mgr.y (mgr.25138) 95 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:47:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:13.870326+0000 mgr.y (mgr.25138) 96 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:47:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:13.885828+0000 mgr.y (mgr.25138) 97 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:47:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:13.893390+0000 mgr.y (mgr.25138) 98 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:13.900532+0000 mgr.y (mgr.25138) 99 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:13.909204+0000 mgr.y (mgr.25138) 100 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:13.909920+0000 mgr.y (mgr.25138) 101 : cephadm [INF] Upgrade: osd.0 is safe to restart 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:13.921204+0000 mgr.y (mgr.25138) 102 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:14.515231+0000 mgr.y (mgr.25138) 103 : audit [DBG] from='client.54216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.769797+0000 mgr.y (mgr.25138) 94 : audit [DBG] from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:13.870248+0000 mgr.y (mgr.25138) 95 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:13.870326+0000 mgr.y (mgr.25138) 96 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:13.885828+0000 mgr.y (mgr.25138) 97 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:13.893390+0000 mgr.y (mgr.25138) 98 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:47:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:13.900532+0000 mgr.y (mgr.25138) 99 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:47:15.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:13.909204+0000 mgr.y (mgr.25138) 100 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T09:47:15.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:13.909920+0000 mgr.y (mgr.25138) 101 : cephadm [INF] Upgrade: osd.0 is safe to restart 2024-02-10T09:47:15.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:13.921204+0000 mgr.y (mgr.25138) 102 : cluster [DBG] pgmap v36: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:15.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:14.515231+0000 mgr.y (mgr.25138) 103 : audit [DBG] from='client.54216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:15.398696+0000 mgr.y (mgr.25138) 104 : cephadm [INF] Upgrade: Updating osd.0 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:15.405588+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:15.406236+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:15.406763+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:15.407785+0000 mgr.y (mgr.25138) 105 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:15.408126+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "container_image"}]: dispatch 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:15.398696+0000 mgr.y (mgr.25138) 104 : cephadm [INF] Upgrade: Updating osd.0 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:15.405588+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:16.628 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:15.406236+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T09:47:16.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:15.406763+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:47:16.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:15.407785+0000 mgr.y (mgr.25138) 105 : cephadm [INF] Deploying daemon osd.0 on smithi153 2024-02-10T09:47:16.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:15.408126+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "container_image"}]: dispatch 2024-02-10T09:47:17.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:15.921607+0000 mgr.y (mgr.25138) 106 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:17.546 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:15.921607+0000 mgr.y (mgr.25138) 106 : cluster [DBG] pgmap v37: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:18.144 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:18 smithi153 systemd[1]: Stopping Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:47:18.421 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:18 smithi153 bash[43214]: debug 2024-02-10T09:47:18.142+0000 7f316715c700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:47:18.421 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:18 smithi153 bash[43214]: debug 2024-02-10T09:47:18.142+0000 7f316715c700 -1 osd.0 73 *** Got signal Terminated *** 2024-02-10T09:47:18.421 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:18 smithi153 bash[43214]: debug 2024-02-10T09:47:18.142+0000 7f316715c700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:47:18.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:18 smithi153 bash[76701]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.0 2024-02-10T09:47:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392142+0000 mon.a (mon.0) 311 : cluster [DBG] osd.0 reported immediately failed by osd.4 2024-02-10T09:47:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392186+0000 mon.a (mon.0) 312 : cluster [INF] osd.0 failed (root=default,host=smithi153) (connection refused reported by osd.4) 2024-02-10T09:47:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392400+0000 mon.a (mon.0) 313 : cluster [DBG] osd.0 reported immediately failed by osd.4 2024-02-10T09:47:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392513+0000 mon.a (mon.0) 314 : cluster [DBG] osd.0 reported immediately failed by osd.1 2024-02-10T09:47:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392590+0000 mon.a (mon.0) 315 : cluster [DBG] osd.0 reported immediately failed by osd.3 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392684+0000 mon.a (mon.0) 316 : cluster [DBG] osd.0 reported immediately failed by osd.5 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392803+0000 mon.a (mon.0) 317 : cluster [DBG] osd.0 reported immediately failed by osd.1 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392875+0000 mon.a (mon.0) 318 : cluster [DBG] osd.0 reported immediately failed by osd.3 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.392971+0000 mon.a (mon.0) 319 : cluster [DBG] osd.0 reported immediately failed by osd.6 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.393100+0000 mon.a (mon.0) 320 : cluster [DBG] osd.0 reported immediately failed by osd.7 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.393233+0000 mon.a (mon.0) 321 : cluster [DBG] osd.0 reported immediately failed by osd.5 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.393365+0000 mon.a (mon.0) 322 : cluster [DBG] osd.0 reported immediately failed by osd.6 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.393496+0000 mon.a (mon.0) 323 : cluster [DBG] osd.0 reported immediately failed by osd.7 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.393626+0000 mon.a (mon.0) 324 : cluster [DBG] osd.0 reported immediately failed by osd.2 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.393763+0000 mon.a (mon.0) 325 : cluster [DBG] osd.0 reported immediately failed by osd.2 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392142+0000 mon.a (mon.0) 311 : cluster [DBG] osd.0 reported immediately failed by osd.4 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392186+0000 mon.a (mon.0) 312 : cluster [INF] osd.0 failed (root=default,host=smithi153) (connection refused reported by osd.4) 2024-02-10T09:47:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392400+0000 mon.a (mon.0) 313 : cluster [DBG] osd.0 reported immediately failed by osd.4 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392513+0000 mon.a (mon.0) 314 : cluster [DBG] osd.0 reported immediately failed by osd.1 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392590+0000 mon.a (mon.0) 315 : cluster [DBG] osd.0 reported immediately failed by osd.3 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392684+0000 mon.a (mon.0) 316 : cluster [DBG] osd.0 reported immediately failed by osd.5 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392803+0000 mon.a (mon.0) 317 : cluster [DBG] osd.0 reported immediately failed by osd.1 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392875+0000 mon.a (mon.0) 318 : cluster [DBG] osd.0 reported immediately failed by osd.3 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.392971+0000 mon.a (mon.0) 319 : cluster [DBG] osd.0 reported immediately failed by osd.6 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.393100+0000 mon.a (mon.0) 320 : cluster [DBG] osd.0 reported immediately failed by osd.7 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.393233+0000 mon.a (mon.0) 321 : cluster [DBG] osd.0 reported immediately failed by osd.5 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.393365+0000 mon.a (mon.0) 322 : cluster [DBG] osd.0 reported immediately failed by osd.6 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.393496+0000 mon.a (mon.0) 323 : cluster [DBG] osd.0 reported immediately failed by osd.7 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.393626+0000 mon.a (mon.0) 324 : cluster [DBG] osd.0 reported immediately failed by osd.2 2024-02-10T09:47:18.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.393763+0000 mon.a (mon.0) 325 : cluster [DBG] osd.0 reported immediately failed by osd.2 2024-02-10T09:47:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:17.921976+0000 mgr.y (mgr.25138) 107 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.411521+0000 mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:47:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:18.423652+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e74: 8 total, 7 up, 8 in 2024-02-10T09:47:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:18.424365+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:47:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:18.426112+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:47:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:17.921976+0000 mgr.y (mgr.25138) 107 : cluster [DBG] pgmap v38: 129 pgs: 129 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.411521+0000 mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:47:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:18.423652+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e74: 8 total, 7 up, 8 in 2024-02-10T09:47:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:18.424365+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T09:47:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:18.426112+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T09:47:20.595 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:19.423103+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:47:20.595 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:19.423248+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e75: 8 total, 7 up, 8 in 2024-02-10T09:47:20.595 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:20 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.0.service: Succeeded. 2024-02-10T09:47:20.595 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:20 smithi153 systemd[1]: Stopped Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:47:20.596 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:19.423103+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T09:47:20.596 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:19.423248+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e75: 8 total, 7 up, 8 in 2024-02-10T09:47:20.939 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:20 smithi153 systemd[1]: Starting Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:47:21.864 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:19.922360+0000 mgr.y (mgr.25138) 108 : cluster [DBG] pgmap v41: 129 pgs: 16 stale+active+clean, 113 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:21.865 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:19.922360+0000 mgr.y (mgr.25138) 108 : cluster [DBG] pgmap v41: 129 pgs: 16 stale+active+clean, 113 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T09:47:21.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:21 smithi153 bash[77006]: --> ceph-volume lvm activate successful for osd ID: 0 2024-02-10T09:47:22.939 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:47:22.572+0000 7fed76bd6700 1 -- 172.21.15.153:0/1501304083 <== mon.1 v2:172.21.15.153:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559696506680 con 0x5596964cd000 2024-02-10T09:47:22.939 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:22 smithi153 bash[77279]: 6c831d5948b5740b779e63b363cf0dc419211bf3cc7667b7036e6ed3f4818664 2024-02-10T09:47:22.939 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:22 smithi153 systemd[1]: Started Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:47:23.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:47:23.093+0000 7fed7e664380 -1 Falling back to public interface 2024-02-10T09:47:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:21.922810+0000 mgr.y (mgr.25138) 109 : cluster [DBG] pgmap v42: 129 pgs: 16 stale+active+clean, 113 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:22.677098+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:22.679792+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:22.680252+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:22.681460+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:23.075328+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:21.922810+0000 mgr.y (mgr.25138) 109 : cluster [DBG] pgmap v42: 129 pgs: 16 stale+active+clean, 113 active+clean; 7.0 KiB data, 6.3 MiB used, 707 GiB / 715 GiB avail 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:22.677098+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:22.679792+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:22.680252+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:22.681460+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:47:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:23.075328+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:47:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:47:24.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:24.445638+0000 mon.a (mon.0) 337 : cluster [WRN] Health check failed: Degraded data redundancy: 66/606 objects degraded (10.891%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:24.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:24.445638+0000 mon.a (mon.0) 337 : cluster [WRN] Health check failed: Degraded data redundancy: 66/606 objects degraded (10.891%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:24] "GET /metrics HTTP/1.1" 200 153050 "" "Prometheus/2.33.4" 2024-02-10T09:47:25.835 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:23.923435+0000 mgr.y (mgr.25138) 110 : cluster [DBG] pgmap v43: 129 pgs: 21 active+undersized+degraded, 28 active+undersized, 80 active+clean; 7.0 KiB data, 6.5 MiB used, 707 GiB / 715 GiB avail; 66/606 objects degraded (10.891%) 2024-02-10T09:47:25.835 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:23.923435+0000 mgr.y (mgr.25138) 110 : cluster [DBG] pgmap v43: 129 pgs: 21 active+undersized+degraded, 28 active+undersized, 80 active+clean; 7.0 KiB data, 6.5 MiB used, 707 GiB / 715 GiB avail; 66/606 objects degraded (10.891%) 2024-02-10T09:47:26.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:47:25.833+0000 7fed7e664380 -1 osd.0 73 log_to_monitors {default=true} 2024-02-10T09:47:26.846 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:47:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:47:26.525+0000 7fed7034c700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:47:26.847 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:25.837353+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/3896183822,v1:172.21.15.153:6803/3896183822]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:47:26.847 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:25.837794+0000 mon.a (mon.0) 338 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:47:26.847 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:25.837353+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/3896183822,v1:172.21.15.153:6803/3896183822]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:47:26.847 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:25.837794+0000 mon.a (mon.0) 338 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T09:47:27.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:47:26.845+0000 7fc4ea3a1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-02-10T09:47:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:25.924122+0000 mgr.y (mgr.25138) 111 : cluster [DBG] pgmap v44: 129 pgs: 21 active+undersized+degraded, 28 active+undersized, 80 active+clean; 7.0 KiB data, 6.5 MiB used, 707 GiB / 715 GiB avail; 66/606 objects degraded (10.891%) 2024-02-10T09:47:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.509043+0000 mon.a (mon.0) 339 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T09:47:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:26.509109+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e76: 8 total, 7 up, 8 in 2024-02-10T09:47:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.509884+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/3896183822,v1:172.21.15.153:6803/3896183822]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.510205+0000 mon.a (mon.0) 341 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.560152+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.837551+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.841721+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.843398+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.844315+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.845079+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:27.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.846046+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:25.924122+0000 mgr.y (mgr.25138) 111 : cluster [DBG] pgmap v44: 129 pgs: 21 active+undersized+degraded, 28 active+undersized, 80 active+clean; 7.0 KiB data, 6.5 MiB used, 707 GiB / 715 GiB avail; 66/606 objects degraded (10.891%) 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.509043+0000 mon.a (mon.0) 339 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:26.509109+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e76: 8 total, 7 up, 8 in 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.509884+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 [v2:172.21.15.153:6802/3896183822,v1:172.21.15.153:6803/3896183822]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.510205+0000 mon.a (mon.0) 341 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.560152+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.837551+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.841721+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.843398+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.844315+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.845079+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:27.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.846046+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:28.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:26.846154+0000 mgr.y (mgr.25138) 112 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:28.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:26.846783+0000 mgr.y (mgr.25138) 113 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-02-10T09:47:28.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:27.506419+0000 mon.a (mon.0) 349 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:27.514130+0000 mon.a (mon.0) 350 : cluster [INF] osd.0 [v2:172.21.15.153:6802/3896183822,v1:172.21.15.153:6803/3896183822] boot 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:27.514190+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:27.519641+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:28.512547+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:26.846154+0000 mgr.y (mgr.25138) 112 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:26.846783+0000 mgr.y (mgr.25138) 113 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:27.506419+0000 mon.a (mon.0) 349 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:27.514130+0000 mon.a (mon.0) 350 : cluster [INF] osd.0 [v2:172.21.15.153:6802/3896183822,v1:172.21.15.153:6803/3896183822] boot 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:27.514190+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:27.519641+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T09:47:28.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:28.512547+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-10T09:47:29.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:27.924794+0000 mgr.y (mgr.25138) 114 : cluster [DBG] pgmap v47: 129 pgs: 21 active+undersized+degraded, 28 active+undersized, 80 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 66/606 objects degraded (10.891%) 2024-02-10T09:47:29.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:27.924794+0000 mgr.y (mgr.25138) 114 : cluster [DBG] pgmap v47: 129 pgs: 21 active+undersized+degraded, 28 active+undersized, 80 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 66/606 objects degraded (10.891%) 2024-02-10T09:47:31.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:29.925428+0000 mgr.y (mgr.25138) 115 : cluster [DBG] pgmap v49: 129 pgs: 33 peering, 7 active+undersized+degraded, 9 active+undersized, 80 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 28/606 objects degraded (4.620%) 2024-02-10T09:47:31.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:30.534508+0000 mon.a (mon.0) 354 : cluster [WRN] Health check update: Degraded data redundancy: 28/606 objects degraded (4.620%), 7 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:29.925428+0000 mgr.y (mgr.25138) 115 : cluster [DBG] pgmap v49: 129 pgs: 33 peering, 7 active+undersized+degraded, 9 active+undersized, 80 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 28/606 objects degraded (4.620%) 2024-02-10T09:47:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:30.534508+0000 mon.a (mon.0) 354 : cluster [WRN] Health check update: Degraded data redundancy: 28/606 objects degraded (4.620%), 7 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:32.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:32.081779+0000 mon.a (mon.0) 355 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/606 objects degraded (4.620%), 7 pgs degraded) 2024-02-10T09:47:32.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:32.081828+0000 mon.a (mon.0) 356 : cluster [INF] Cluster is now healthy 2024-02-10T09:47:32.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:32.081779+0000 mon.a (mon.0) 355 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/606 objects degraded (4.620%), 7 pgs degraded) 2024-02-10T09:47:32.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:32.081828+0000 mon.a (mon.0) 356 : cluster [INF] Cluster is now healthy 2024-02-10T09:47:33.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:31.925915+0000 mgr.y (mgr.25138) 116 : cluster [DBG] pgmap v50: 129 pgs: 33 peering, 96 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:33.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:31.925915+0000 mgr.y (mgr.25138) 116 : cluster [DBG] pgmap v50: 129 pgs: 33 peering, 96 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:47:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:47:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:34] "GET /metrics HTTP/1.1" 200 153050 "" "Prometheus/2.33.4" 2024-02-10T09:47:35.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:33.926573+0000 mgr.y (mgr.25138) 117 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:35.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:33.926573+0000 mgr.y (mgr.25138) 117 : cluster [DBG] pgmap v51: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:37.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:35.927178+0000 mgr.y (mgr.25138) 118 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:37.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:35.927178+0000 mgr.y (mgr.25138) 118 : cluster [DBG] pgmap v52: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:37.927761+0000 mgr.y (mgr.25138) 119 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:38.076238+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:38.118220+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:39.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:37.927761+0000 mgr.y (mgr.25138) 119 : cluster [DBG] pgmap v53: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:38.076238+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:38.118220+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:39.928584+0000 mgr.y (mgr.25138) 120 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:41.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:39.928584+0000 mgr.y (mgr.25138) 120 : cluster [DBG] pgmap v54: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:42.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:41.861297+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:42.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:41.861297+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:43.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:41.861480+0000 mgr.y (mgr.25138) 121 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:43.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:41.862760+0000 mgr.y (mgr.25138) 122 : cephadm [INF] Upgrade: osd.1 is safe to restart 2024-02-10T09:47:43.173 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:41.928993+0000 mgr.y (mgr.25138) 123 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:43.173 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:41.861480+0000 mgr.y (mgr.25138) 121 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T09:47:43.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:41.862760+0000 mgr.y (mgr.25138) 122 : cephadm [INF] Upgrade: osd.1 is safe to restart 2024-02-10T09:47:43.174 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:41.928993+0000 mgr.y (mgr.25138) 123 : cluster [DBG] pgmap v55: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:47:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:47:44.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:43.190548+0000 mgr.y (mgr.25138) 124 : cephadm [INF] Upgrade: Updating osd.1 2024-02-10T09:47:44.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:43.196593+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:44.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:43.197214+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:43.197696+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:43.198719+0000 mgr.y (mgr.25138) 125 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:43.199054+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "container_image"}]: dispatch 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:43.190548+0000 mgr.y (mgr.25138) 124 : cephadm [INF] Upgrade: Updating osd.1 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:43.196593+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:43.197214+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:43.197696+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:43.198719+0000 mgr.y (mgr.25138) 125 : cephadm [INF] Deploying daemon osd.1 on smithi153 2024-02-10T09:47:44.605 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:43.199054+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "container_image"}]: dispatch 2024-02-10T09:47:44.606 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:44] "GET /metrics HTTP/1.1" 200 161355 "" "Prometheus/2.33.4" 2024-02-10T09:47:44.853 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (15m) 18s ago 16m 26.1M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (3m) 92s ago 18m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (4m) 92s ago 29m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (11m) 18s ago 30m 426M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (110s) 18s ago 31m 61.8M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:47:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (2m) 92s ago 29m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (87s) 18s ago 29m 59.7M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (3m) 18s ago 21m 14.7M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (3m) 92s ago 21m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (23s) 18s ago 28m 11.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (28m) 18s ago 28m 45.4M 4096M 15.2.0 204a01f9b0b6 d213cd3c6620 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (28m) 18s ago 28m 44.1M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (27m) 18s ago 27m 47.4M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (27m) 92s ago 27m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (27m) 92s ago 27m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (26m) 92s ago 26m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (26m) 92s ago 26m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (15m) 92s ago 15m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (18m) 18s ago 18m 37.3M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:47:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (18m) 92s ago 18m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:47:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:43.929643+0000 mgr.y (mgr.25138) 126 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:45.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:43.929643+0000 mgr.y (mgr.25138) 126 : cluster [DBG] pgmap v56: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:47:45.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 7, 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 1 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9, 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 6 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:47:45.890 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:47:46.234 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "1/8 daemons upgraded", 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:47:46.235 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:47:46.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:44.841070+0000 mgr.y (mgr.25138) 127 : audit [DBG] from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.170394+0000 mgr.y (mgr.25138) 128 : audit [DBG] from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.488018+0000 mgr.y (mgr.25138) 129 : audit [DBG] from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.489807+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.490165+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.490424+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.490690+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.490962+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.491209+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.492794+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.493073+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.493329+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.493576+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.493818+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:45.888514+0000 mon.a (mon.0) 375 : audit [DBG] from='client.54246 172.21.15.153:0/1935207768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:46.306 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:46 smithi153 systemd[1]: Stopping Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:47:46.306 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:44.841070+0000 mgr.y (mgr.25138) 127 : audit [DBG] from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:46.306 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.170394+0000 mgr.y (mgr.25138) 128 : audit [DBG] from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:46.306 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.488018+0000 mgr.y (mgr.25138) 129 : audit [DBG] from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:46.306 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.489807+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:46.306 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.490165+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.490424+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.490690+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.490962+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.491209+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.492794+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.493073+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.493329+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.493576+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.493818+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:47:46.307 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:45.888514+0000 mon.a (mon.0) 375 : audit [DBG] from='client.54246 172.21.15.153:0/1935207768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:46.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:46 smithi153 bash[45199]: debug 2024-02-10T09:47:46.301+0000 7f95c5f26700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:47:46.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:46 smithi153 bash[45199]: debug 2024-02-10T09:47:46.301+0000 7f95c5f26700 -1 osd.1 78 *** Got signal Terminated *** 2024-02-10T09:47:46.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:46 smithi153 bash[45199]: debug 2024-02-10T09:47:46.301+0000 7f95c5f26700 -1 osd.1 78 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:47:47.004 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:46 smithi153 bash[78471]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.1 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:45.929939+0000 mgr.y (mgr.25138) 130 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:46.233091+0000 mgr.y (mgr.25138) 131 : audit [DBG] from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.474548+0000 mon.a (mon.0) 376 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.474581+0000 mon.a (mon.0) 377 : cluster [INF] osd.1 failed (root=default,host=smithi153) (connection refused reported by osd.3) 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.474902+0000 mon.a (mon.0) 378 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.475209+0000 mon.a (mon.0) 379 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.475891+0000 mon.a (mon.0) 380 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.553817+0000 mon.a (mon.0) 381 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.553945+0000 mon.a (mon.0) 382 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554130+0000 mon.a (mon.0) 383 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554207+0000 mon.a (mon.0) 384 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554371+0000 mon.a (mon.0) 385 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554458+0000 mon.a (mon.0) 386 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554566+0000 mon.a (mon.0) 387 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554690+0000 mon.a (mon.0) 388 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554809+0000 mon.a (mon.0) 389 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.554934+0000 mon.a (mon.0) 390 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.555031+0000 mon.a (mon.0) 391 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.555134+0000 mon.a (mon.0) 392 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.555226+0000 mon.a (mon.0) 393 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.555331+0000 mon.a (mon.0) 394 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.589896+0000 mon.a (mon.0) 395 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.675028+0000 mon.a (mon.0) 396 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.675708+0000 mon.a (mon.0) 397 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.675795+0000 mon.a (mon.0) 398 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.676653+0000 mon.a (mon.0) 399 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.790889+0000 mon.a (mon.0) 400 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955095+0000 mon.a (mon.0) 401 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955270+0000 mon.a (mon.0) 402 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955363+0000 mon.a (mon.0) 403 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955473+0000 mon.a (mon.0) 404 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955612+0000 mon.a (mon.0) 405 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955687+0000 mon.a (mon.0) 406 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955814+0000 mon.a (mon.0) 407 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.955950+0000 mon.a (mon.0) 408 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.956051+0000 mon.a (mon.0) 409 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.956152+0000 mon.a (mon.0) 410 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.956249+0000 mon.a (mon.0) 411 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.956349+0000 mon.a (mon.0) 412 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.956456+0000 mon.a (mon.0) 413 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:46.956545+0000 mon.a (mon.0) 414 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.076094+0000 mon.a (mon.0) 415 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.076755+0000 mon.a (mon.0) 416 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.076842+0000 mon.a (mon.0) 417 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.077657+0000 mon.a (mon.0) 418 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.191908+0000 mon.a (mon.0) 419 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:45.929939+0000 mgr.y (mgr.25138) 130 : cluster [DBG] pgmap v57: 129 pgs: 129 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:46.233091+0000 mgr.y (mgr.25138) 131 : audit [DBG] from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:47:47.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.474548+0000 mon.a (mon.0) 376 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.474581+0000 mon.a (mon.0) 377 : cluster [INF] osd.1 failed (root=default,host=smithi153) (connection refused reported by osd.3) 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.474902+0000 mon.a (mon.0) 378 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.475209+0000 mon.a (mon.0) 379 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.475891+0000 mon.a (mon.0) 380 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.553817+0000 mon.a (mon.0) 381 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.553945+0000 mon.a (mon.0) 382 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554130+0000 mon.a (mon.0) 383 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554207+0000 mon.a (mon.0) 384 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554371+0000 mon.a (mon.0) 385 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554458+0000 mon.a (mon.0) 386 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554566+0000 mon.a (mon.0) 387 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554690+0000 mon.a (mon.0) 388 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554809+0000 mon.a (mon.0) 389 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.554934+0000 mon.a (mon.0) 390 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.555031+0000 mon.a (mon.0) 391 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.555134+0000 mon.a (mon.0) 392 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.555226+0000 mon.a (mon.0) 393 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.555331+0000 mon.a (mon.0) 394 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.589896+0000 mon.a (mon.0) 395 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.675028+0000 mon.a (mon.0) 396 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.675708+0000 mon.a (mon.0) 397 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.675795+0000 mon.a (mon.0) 398 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.676653+0000 mon.a (mon.0) 399 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.790889+0000 mon.a (mon.0) 400 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955095+0000 mon.a (mon.0) 401 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955270+0000 mon.a (mon.0) 402 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955363+0000 mon.a (mon.0) 403 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955473+0000 mon.a (mon.0) 404 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955612+0000 mon.a (mon.0) 405 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955687+0000 mon.a (mon.0) 406 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955814+0000 mon.a (mon.0) 407 : cluster [DBG] osd.1 reported immediately failed by osd.0 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.955950+0000 mon.a (mon.0) 408 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.956051+0000 mon.a (mon.0) 409 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.956152+0000 mon.a (mon.0) 410 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.956249+0000 mon.a (mon.0) 411 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.956349+0000 mon.a (mon.0) 412 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.956456+0000 mon.a (mon.0) 413 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:46.956545+0000 mon.a (mon.0) 414 : cluster [DBG] osd.1 reported immediately failed by osd.2 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.076094+0000 mon.a (mon.0) 415 : cluster [DBG] osd.1 reported immediately failed by osd.3 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.076755+0000 mon.a (mon.0) 416 : cluster [DBG] osd.1 reported immediately failed by osd.7 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.076842+0000 mon.a (mon.0) 417 : cluster [DBG] osd.1 reported immediately failed by osd.5 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.077657+0000 mon.a (mon.0) 418 : cluster [DBG] osd.1 reported immediately failed by osd.6 2024-02-10T09:47:47.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.191908+0000 mon.a (mon.0) 419 : cluster [DBG] osd.1 reported immediately failed by osd.4 2024-02-10T09:47:48.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.210580+0000 mon.a (mon.0) 420 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:47:48.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.222345+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e79: 8 total, 7 up, 8 in 2024-02-10T09:47:48.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.210580+0000 mon.a (mon.0) 420 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:47:48.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.222345+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e79: 8 total, 7 up, 8 in 2024-02-10T09:47:48.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:48 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.1.service: Succeeded. 2024-02-10T09:47:48.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:48 smithi153 systemd[1]: Stopped Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:47:48.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:48 smithi153 systemd[1]: Starting Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:47:49.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:47.930293+0000 mgr.y (mgr.25138) 132 : cluster [DBG] pgmap v59: 129 pgs: 18 stale+active+clean, 111 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:49.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:48.230066+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e80: 8 total, 7 up, 8 in 2024-02-10T09:47:49.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:47.930293+0000 mgr.y (mgr.25138) 132 : cluster [DBG] pgmap v59: 129 pgs: 18 stale+active+clean, 111 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:49.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:48.230066+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e80: 8 total, 7 up, 8 in 2024-02-10T09:47:49.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:47:49.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T09:47:49.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-02-10T09:47:49.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T09:47:49.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-02-10T09:47:49.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T09:47:49.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T09:47:49.940 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:49 smithi153 bash[78774]: --> ceph-volume lvm activate successful for osd ID: 1 2024-02-10T09:47:50.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:47:50.672+0000 7f68390f3700 1 -- 172.21.15.153:0/1492046049 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ff8086680 con 0x559ff804d800 2024-02-10T09:47:50.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:50 smithi153 bash[79042]: 24b6f232a23ef62cf13dffba4ea26c7395cade7093290dc927ec25866f7fd302 2024-02-10T09:47:50.939 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:50 smithi153 systemd[1]: Started Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:47:51.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:47:51.192+0000 7f6840b81380 -1 Falling back to public interface 2024-02-10T09:47:51.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:49.931137+0000 mgr.y (mgr.25138) 133 : cluster [DBG] pgmap v61: 129 pgs: 51 peering, 1 stale+active+clean, 77 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:50.226268+0000 mon.a (mon.0) 423 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:50.788990+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:50.792195+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:50.792672+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:50.793931+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:49.931137+0000 mgr.y (mgr.25138) 133 : cluster [DBG] pgmap v61: 129 pgs: 51 peering, 1 stale+active+clean, 77 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:50.226268+0000 mon.a (mon.0) 423 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:50.788990+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:50.792195+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:50.792672+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:51.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:50.793931+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:47:52.396 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:52.085410+0000 mon.a (mon.0) 428 : cluster [WRN] Health check failed: Degraded data redundancy: 7/606 objects degraded (1.155%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:52.397 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:52.085410+0000 mon.a (mon.0) 428 : cluster [WRN] Health check failed: Degraded data redundancy: 7/606 objects degraded (1.155%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:51.931788+0000 mgr.y (mgr.25138) 134 : cluster [DBG] pgmap v62: 129 pgs: 2 active+undersized+degraded, 51 peering, 3 active+undersized, 73 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 7/606 objects degraded (1.155%) 2024-02-10T09:47:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:53.075930+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:53.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:51.931788+0000 mgr.y (mgr.25138) 134 : cluster [DBG] pgmap v62: 129 pgs: 2 active+undersized+degraded, 51 peering, 3 active+undersized, 73 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 7/606 objects degraded (1.155%) 2024-02-10T09:47:53.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:53.075930+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:47:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:47:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:53.941356+0000 mon.c (mon.1) 8 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/2208511089,v1:172.21.15.153:6811/2208511089]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:47:54.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:53.941738+0000 mon.a (mon.0) 430 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:47:54.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:47:53.936+0000 7f6840b81380 -1 osd.1 78 log_to_monitors {default=true} 2024-02-10T09:47:54.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:47:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:47:54.166+0000 7f683306a700 -1 osd.1 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:47:54.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:53.941356+0000 mon.c (mon.1) 8 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/2208511089,v1:172.21.15.153:6811/2208511089]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:47:54.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:53.941738+0000 mon.a (mon.0) 430 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T09:47:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:47:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:47:54] "GET /metrics HTTP/1.1" 200 161516 "" "Prometheus/2.33.4" 2024-02-10T09:47:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:53.932416+0000 mgr.y (mgr.25138) 135 : cluster [DBG] pgmap v63: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 84/606 objects degraded (13.861%) 2024-02-10T09:47:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:54.139260+0000 mon.a (mon.0) 431 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 14 pgs peering) 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.145846+0000 mon.a (mon.0) 432 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:54.145908+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e81: 8 total, 7 up, 8 in 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.148115+0000 mon.c (mon.1) 9 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/2208511089,v1:172.21.15.153:6811/2208511089]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.149387+0000 mon.a (mon.0) 434 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.430800+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.437224+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.441434+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.443290+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.444689+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.445672+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.446678+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.447677+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:54.448208+0000 mgr.y (mgr.25138) 136 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:53.932416+0000 mgr.y (mgr.25138) 135 : cluster [DBG] pgmap v63: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 297 MiB used, 708 GiB / 715 GiB avail; 84/606 objects degraded (13.861%) 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:54.139260+0000 mon.a (mon.0) 431 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 14 pgs peering) 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.145846+0000 mon.a (mon.0) 432 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:54.145908+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e81: 8 total, 7 up, 8 in 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.148115+0000 mon.c (mon.1) 9 : audit [INF] from='osd.1 [v2:172.21.15.153:6810/2208511089,v1:172.21.15.153:6811/2208511089]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.149387+0000 mon.a (mon.0) 434 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.430800+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.437224+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.441434+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:55.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.443290+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.444689+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.445672+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.446678+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.447677+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:54.448208+0000 mgr.y (mgr.25138) 136 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.448792+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.454249+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.454812+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.460075+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.460641+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.466099+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:47:55.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.466660+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.467109+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.467544+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.468003+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.468427+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.468897+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.469320+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.469775+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.448792+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.454249+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.454812+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.460075+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:47:55.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.460641+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.466099+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.466660+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.467109+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.467544+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.468003+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.468427+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.468897+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.469320+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.469775+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.444 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.470198+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.470615+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.471034+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.471450+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.476873+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:47:54.477327+0000 mgr.y (mgr.25138) 137 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.477654+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.482106+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.482498+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.489013+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.492698+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:54.497062+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.470198+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.470615+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.471034+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.471450+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.476873+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:47:54.477327+0000 mgr.y (mgr.25138) 137 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.477654+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.482106+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.482498+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.489013+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.492698+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:47:55.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:54.497062+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:55.143951+0000 mon.a (mon.0) 468 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:47:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:55.149456+0000 mon.a (mon.0) 469 : cluster [INF] osd.1 [v2:172.21.15.153:6810/2208511089,v1:172.21.15.153:6811/2208511089] boot 2024-02-10T09:47:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:55.149486+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-10T09:47:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:55.150473+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:47:56.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:56.153141+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-10T09:47:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:55.143951+0000 mon.a (mon.0) 468 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:47:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:55.149456+0000 mon.a (mon.0) 469 : cluster [INF] osd.1 [v2:172.21.15.153:6810/2208511089,v1:172.21.15.153:6811/2208511089] boot 2024-02-10T09:47:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:55.149486+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-10T09:47:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:55.150473+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T09:47:56.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:56.153141+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-10T09:47:57.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:55.933100+0000 mgr.y (mgr.25138) 138 : cluster [DBG] pgmap v66: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 587 MiB used, 709 GiB / 715 GiB avail; 84/606 objects degraded (13.861%) 2024-02-10T09:47:57.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:55.933100+0000 mgr.y (mgr.25138) 138 : cluster [DBG] pgmap v66: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 587 MiB used, 709 GiB / 715 GiB avail; 84/606 objects degraded (13.861%) 2024-02-10T09:47:59.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:57.933833+0000 mgr.y (mgr.25138) 139 : cluster [DBG] pgmap v68: 129 pgs: 21 active+undersized+degraded, 31 active+undersized, 77 active+clean; 7.0 KiB data, 587 MiB used, 709 GiB / 715 GiB avail; 78/606 objects degraded (12.871%) 2024-02-10T09:47:59.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:47:58.130695+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:59.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:47:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:58.173762+0000 mon.a (mon.0) 474 : cluster [WRN] Health check update: Degraded data redundancy: 78/606 objects degraded (12.871%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T09:47:59.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:57.933833+0000 mgr.y (mgr.25138) 139 : cluster [DBG] pgmap v68: 129 pgs: 21 active+undersized+degraded, 31 active+undersized, 77 active+clean; 7.0 KiB data, 587 MiB used, 709 GiB / 715 GiB avail; 78/606 objects degraded (12.871%) 2024-02-10T09:47:59.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:47:58.130695+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:47:59.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:47:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:58.173762+0000 mon.a (mon.0) 474 : cluster [WRN] Health check update: Degraded data redundancy: 78/606 objects degraded (12.871%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T09:48:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:47:59.934619+0000 mgr.y (mgr.25138) 140 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:00.130049+0000 mon.a (mon.0) 475 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/606 objects degraded (12.871%), 21 pgs degraded) 2024-02-10T09:48:01.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:00.130083+0000 mon.a (mon.0) 476 : cluster [INF] Cluster is now healthy 2024-02-10T09:48:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:47:59.934619+0000 mgr.y (mgr.25138) 140 : cluster [DBG] pgmap v69: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:00.130049+0000 mon.a (mon.0) 475 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/606 objects degraded (12.871%), 21 pgs degraded) 2024-02-10T09:48:01.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:00.130083+0000 mon.a (mon.0) 476 : cluster [INF] Cluster is now healthy 2024-02-10T09:48:03.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:01.935019+0000 mgr.y (mgr.25138) 141 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:03.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:01.935019+0000 mgr.y (mgr.25138) 141 : cluster [DBG] pgmap v70: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:48:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:48:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:03.138004+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:03.138004+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:48:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:04] "GET /metrics HTTP/1.1" 200 161516 "" "Prometheus/2.33.4" 2024-02-10T09:48:05.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:03.935658+0000 mgr.y (mgr.25138) 142 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:05.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:03.935658+0000 mgr.y (mgr.25138) 142 : cluster [DBG] pgmap v71: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:07.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:05.936115+0000 mgr.y (mgr.25138) 143 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:07.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:05.936115+0000 mgr.y (mgr.25138) 143 : cluster [DBG] pgmap v72: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:07.926745+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:48:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:07.927001+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:48:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:07.927282+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:48:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:07.927551+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:48:08.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:08.076772+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:07.926745+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-02-10T09:48:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:07.927001+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-10T09:48:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:07.927282+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T09:48:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:07.927551+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:48:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:08.076772+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:07.936564+0000 mgr.y (mgr.25138) 144 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:09.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:07.936564+0000 mgr.y (mgr.25138) 144 : cluster [DBG] pgmap v73: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:09.937367+0000 mgr.y (mgr.25138) 145 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:09.937367+0000 mgr.y (mgr.25138) 145 : cluster [DBG] pgmap v74: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:13.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:11.937939+0000 mgr.y (mgr.25138) 146 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:13.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:11.937939+0000 mgr.y (mgr.25138) 146 : cluster [DBG] pgmap v75: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:48:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:48:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:48:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:14] "GET /metrics HTTP/1.1" 200 162246 "" "Prometheus/2.33.4" 2024-02-10T09:48:15.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:13.938545+0000 mgr.y (mgr.25138) 147 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:15.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:13.938545+0000 mgr.y (mgr.25138) 147 : cluster [DBG] pgmap v76: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:17.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:48:17.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:15.939204+0000 mgr.y (mgr.25138) 148 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:17.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:16.562479+0000 mgr.y (mgr.25138) 149 : audit [DBG] from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:17.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:15.939204+0000 mgr.y (mgr.25138) 148 : cluster [DBG] pgmap v77: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:17.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:16.562479+0000 mgr.y (mgr.25138) 149 : audit [DBG] from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:19.133 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (16m) 24s ago 17m 26.3M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (4m) 2m ago 19m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (4m) 2m ago 29m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (12m) 24s ago 31m 430M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (2m) 24s ago 31m 64.8M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (3m) 2m ago 29m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (2m) 24s ago 30m 58.1M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (4m) 24s ago 22m 15.1M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (4m) 2m ago 22m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (56s) 24s ago 29m 37.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (28s) 24s ago 29m 10.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:48:19.134 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (28m) 24s ago 28m 43.9M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (28m) 24s ago 28m 48.6M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (28m) 2m ago 28m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (27m) 2m ago 27m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (27m) 2m ago 27m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (27m) 2m ago 27m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (16m) 2m ago 16m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (19m) 24s ago 19m 37.5M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:48:19.135 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (19m) 2m ago 19m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:17.939588+0000 mgr.y (mgr.25138) 150 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.124352+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.124674+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.124928+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.125175+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.125420+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.125655+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.127157+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.127411+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.127646+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.127884+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.128111+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:17.939588+0000 mgr.y (mgr.25138) 150 : cluster [DBG] pgmap v78: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.124352+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.124674+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.124928+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.125175+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.125420+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.125655+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.127157+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.127411+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.127646+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.127884+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.128111+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:19.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-02-10T09:48:20.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:19.122484+0000 mgr.y (mgr.25138) 151 : audit [DBG] from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:20.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:19.122484+0000 mgr.y (mgr.25138) 151 : audit [DBG] from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:21.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:19.940211+0000 mgr.y (mgr.25138) 152 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:21.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:19.940211+0000 mgr.y (mgr.25138) 152 : cluster [DBG] pgmap v79: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:21.919 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:48:22.267 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:21.907300+0000 mon.a (mon.0) 494 : audit [DBG] from='client.54276 172.21.15.153:0/1036650710' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:22.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:21.907300+0000 mon.a (mon.0) 494 : audit [DBG] from='client.54276 172.21.15.153:0/1036650710' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:22.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-02-10T09:48:23.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:21.940793+0000 mgr.y (mgr.25138) 153 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:23.542 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:21.940793+0000 mgr.y (mgr.25138) 153 : cluster [DBG] pgmap v80: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:48:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:48:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:48:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:24] "GET /metrics HTTP/1.1" 200 162247 "" "Prometheus/2.33.4" 2024-02-10T09:48:25.634 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:23.941395+0000 mgr.y (mgr.25138) 154 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:25.635 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:24.443968+0000 mgr.y (mgr.25138) 155 : audit [DBG] from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:25.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:23.941395+0000 mgr.y (mgr.25138) 154 : cluster [DBG] pgmap v81: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:25.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:24.443968+0000 mgr.y (mgr.25138) 155 : audit [DBG] from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:26.238 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:48:26.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-02-10T09:48:27.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:25.941836+0000 mgr.y (mgr.25138) 156 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:27.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:25.941836+0000 mgr.y (mgr.25138) 156 : cluster [DBG] pgmap v82: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:29.662 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:27.942319+0000 mgr.y (mgr.25138) 157 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:29.662 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:27.942319+0000 mgr.y (mgr.25138) 157 : cluster [DBG] pgmap v83: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:30.522 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:28.728412+0000 mgr.y (mgr.25138) 158 : audit [DBG] from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:30.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:28.728412+0000 mgr.y (mgr.25138) 158 : audit [DBG] from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:30.543 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:48:31.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:48:31.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:29.942929+0000 mgr.y (mgr.25138) 159 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:30.539094+0000 mgr.y (mgr.25138) 160 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:30.542680+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:30.547550+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:30.847371+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:29.942929+0000 mgr.y (mgr.25138) 159 : cluster [DBG] pgmap v84: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:30.539094+0000 mgr.y (mgr.25138) 160 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:30.542680+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:30.547550+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:30.847371+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:32.828 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:30.850964+0000 mgr.y (mgr.25138) 161 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:48:32.828 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:30.850964+0000 mgr.y (mgr.25138) 161 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:48:33.639 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:48:33.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:31.943309+0000 mgr.y (mgr.25138) 162 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:33.814 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:31.943309+0000 mgr.y (mgr.25138) 162 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:48:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:48:34.294 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (16m) 39s ago 17m 26.3M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (4m) 2m ago 19m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (4m) 2m ago 29m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (12m) 39s ago 31m 430M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (2m) 39s ago 31m 64.8M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (3m) 2m ago 30m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (2m) 39s ago 30m 58.1M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (4m) 39s ago 22m 15.1M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (4m) 2m ago 22m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (72s) 39s ago 29m 37.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (44s) 39s ago 29m 10.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:48:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (28m) 39s ago 28m 43.9M 4096M 15.2.0 204a01f9b0b6 fd3d7b02f03d 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (28m) 39s ago 28m 48.6M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (28m) 2m ago 28m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (28m) 2m ago 28m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (27m) 2m ago 27m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (27m) 2m ago 27m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (16m) 2m ago 16m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (19m) 39s ago 19m 37.5M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:48:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (19m) 2m ago 19m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:48:34.634 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:34] "GET /metrics HTTP/1.1" 200 162247 "" "Prometheus/2.33.4" 2024-02-10T09:48:34.689 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:48:34.689 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:48:34.689 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:48:34.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:48:34.689 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 6, 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8, 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 7 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:48:34.690 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:48:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.627242+0000 mgr.y (mgr.25138) 163 : audit [DBG] from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.835084+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:33.835395+0000 mgr.y (mgr.25138) 164 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:33.835451+0000 mgr.y (mgr.25138) 165 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.836446+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.840805+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.841952+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:33.842519+0000 mgr.y (mgr.25138) 166 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.847952+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.849054+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:33.849614+0000 mgr.y (mgr.25138) 167 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.854146+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.855039+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:33.855596+0000 mgr.y (mgr.25138) 168 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.860107+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.861274+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.627242+0000 mgr.y (mgr.25138) 163 : audit [DBG] from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.835084+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:33.835395+0000 mgr.y (mgr.25138) 164 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:33.835451+0000 mgr.y (mgr.25138) 165 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.836446+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.840805+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.841952+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:33.842519+0000 mgr.y (mgr.25138) 166 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.847952+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.849054+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:33.849614+0000 mgr.y (mgr.25138) 167 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.854146+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.855039+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:33.855596+0000 mgr.y (mgr.25138) 168 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.860107+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:34.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.861274+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.861399+0000 mgr.y (mgr.25138) 169 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:33.862087+0000 mgr.y (mgr.25138) 170 : cephadm [INF] Upgrade: osd.2 is safe to restart 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:33.943849+0000 mgr.y (mgr.25138) 171 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:33.957687+0000 mgr.y (mgr.25138) 172 : audit [DBG] from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.283619+0000 mgr.y (mgr.25138) 173 : audit [DBG] from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.285462+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.285819+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.286105+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.286384+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.286633+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.286897+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.288465+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.288740+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.289004+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.289254+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.289498+0000 mon.a (mon.0) 518 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:34.688871+0000 mon.a (mon.0) 519 : audit [DBG] from='client.54312 172.21.15.153:0/2611691207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.861399+0000 mgr.y (mgr.25138) 169 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:33.862087+0000 mgr.y (mgr.25138) 170 : cephadm [INF] Upgrade: osd.2 is safe to restart 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:33.943849+0000 mgr.y (mgr.25138) 171 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:34.944 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:33.957687+0000 mgr.y (mgr.25138) 172 : audit [DBG] from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.283619+0000 mgr.y (mgr.25138) 173 : audit [DBG] from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.285462+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.285819+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.286105+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.286384+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.286633+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.286897+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.288465+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.288740+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.289004+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.289254+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.289498+0000 mon.a (mon.0) 518 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:48:34.945 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:34.688871+0000 mon.a (mon.0) 519 : audit [DBG] from='client.54312 172.21.15.153:0/2611691207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "2/8 daemons upgraded", 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:48:35.013 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:48:36.508 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:35.012526+0000 mgr.y (mgr.25138) 174 : audit [DBG] from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:36.508 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:35.286222+0000 mgr.y (mgr.25138) 175 : cephadm [INF] Upgrade: Updating osd.2 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:35.292449+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:35.293068+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:35.293587+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:35.294624+0000 mgr.y (mgr.25138) 176 : cephadm [INF] Deploying daemon osd.2 on smithi153 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:35.294959+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "container_image"}]: dispatch 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:35.012526+0000 mgr.y (mgr.25138) 174 : audit [DBG] from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:35.286222+0000 mgr.y (mgr.25138) 175 : cephadm [INF] Upgrade: Updating osd.2 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:35.292449+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:35.293068+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:35.293587+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:35.294624+0000 mgr.y (mgr.25138) 176 : cephadm [INF] Deploying daemon osd.2 on smithi153 2024-02-10T09:48:36.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:35.294959+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "container_image"}]: dispatch 2024-02-10T09:48:37.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:35.944219+0000 mgr.y (mgr.25138) 177 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:37.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:35.944219+0000 mgr.y (mgr.25138) 177 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:38.045 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:37 smithi153 systemd[1]: Stopping Ceph osd.2 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:48:38.306 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:38 smithi153 bash[45769]: debug 2024-02-10T09:48:38.043+0000 7f14fdf42700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:48:38.306 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:38 smithi153 bash[45769]: debug 2024-02-10T09:48:38.043+0000 7f14fdf42700 -1 osd.2 83 *** Got signal Terminated *** 2024-02-10T09:48:38.307 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:38 smithi153 bash[45769]: debug 2024-02-10T09:48:38.043+0000 7f14fdf42700 -1 osd.2 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:48:38.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:38 smithi153 bash[81401]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.2 2024-02-10T09:48:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.089329+0000 mon.a (mon.0) 524 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.089368+0000 mon.a (mon.0) 525 : cluster [INF] osd.2 failed (root=default,host=smithi153) (connection refused reported by osd.1) 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.089594+0000 mon.a (mon.0) 526 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.289078+0000 mon.a (mon.0) 527 : cluster [DBG] osd.2 reported immediately failed by osd.3 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290028+0000 mon.a (mon.0) 528 : cluster [DBG] osd.2 reported immediately failed by osd.5 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290203+0000 mon.a (mon.0) 529 : cluster [DBG] osd.2 reported immediately failed by osd.6 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290280+0000 mon.a (mon.0) 530 : cluster [DBG] osd.2 reported immediately failed by osd.3 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290477+0000 mon.a (mon.0) 531 : cluster [DBG] osd.2 reported immediately failed by osd.0 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290621+0000 mon.a (mon.0) 532 : cluster [DBG] osd.2 reported immediately failed by osd.7 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290698+0000 mon.a (mon.0) 533 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290865+0000 mon.a (mon.0) 534 : cluster [DBG] osd.2 reported immediately failed by osd.4 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.290937+0000 mon.a (mon.0) 535 : cluster [DBG] osd.2 reported immediately failed by osd.0 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.291087+0000 mon.a (mon.0) 536 : cluster [DBG] osd.2 reported immediately failed by osd.5 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.291195+0000 mon.a (mon.0) 537 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.291333+0000 mon.a (mon.0) 538 : cluster [DBG] osd.2 reported immediately failed by osd.6 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.291446+0000 mon.a (mon.0) 539 : cluster [DBG] osd.2 reported immediately failed by osd.7 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.291545+0000 mon.a (mon.0) 540 : cluster [DBG] osd.2 reported immediately failed by osd.4 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.089329+0000 mon.a (mon.0) 524 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.089368+0000 mon.a (mon.0) 525 : cluster [INF] osd.2 failed (root=default,host=smithi153) (connection refused reported by osd.1) 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.089594+0000 mon.a (mon.0) 526 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.289078+0000 mon.a (mon.0) 527 : cluster [DBG] osd.2 reported immediately failed by osd.3 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290028+0000 mon.a (mon.0) 528 : cluster [DBG] osd.2 reported immediately failed by osd.5 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290203+0000 mon.a (mon.0) 529 : cluster [DBG] osd.2 reported immediately failed by osd.6 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290280+0000 mon.a (mon.0) 530 : cluster [DBG] osd.2 reported immediately failed by osd.3 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290477+0000 mon.a (mon.0) 531 : cluster [DBG] osd.2 reported immediately failed by osd.0 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290621+0000 mon.a (mon.0) 532 : cluster [DBG] osd.2 reported immediately failed by osd.7 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290698+0000 mon.a (mon.0) 533 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290865+0000 mon.a (mon.0) 534 : cluster [DBG] osd.2 reported immediately failed by osd.4 2024-02-10T09:48:38.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.290937+0000 mon.a (mon.0) 535 : cluster [DBG] osd.2 reported immediately failed by osd.0 2024-02-10T09:48:38.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.291087+0000 mon.a (mon.0) 536 : cluster [DBG] osd.2 reported immediately failed by osd.5 2024-02-10T09:48:38.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.291195+0000 mon.a (mon.0) 537 : cluster [DBG] osd.2 reported immediately failed by osd.1 2024-02-10T09:48:38.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.291333+0000 mon.a (mon.0) 538 : cluster [DBG] osd.2 reported immediately failed by osd.6 2024-02-10T09:48:38.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.291446+0000 mon.a (mon.0) 539 : cluster [DBG] osd.2 reported immediately failed by osd.7 2024-02-10T09:48:38.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.291545+0000 mon.a (mon.0) 540 : cluster [DBG] osd.2 reported immediately failed by osd.4 2024-02-10T09:48:39.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:37.944623+0000 mgr.y (mgr.25138) 178 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:39.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.299378+0000 mon.a (mon.0) 541 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:48:39.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:38.310129+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e84: 8 total, 7 up, 8 in 2024-02-10T09:48:39.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:37.944623+0000 mgr.y (mgr.25138) 178 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:39.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.299378+0000 mon.a (mon.0) 541 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:48:39.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:38.310129+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e84: 8 total, 7 up, 8 in 2024-02-10T09:48:40.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:39.308563+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e85: 8 total, 7 up, 8 in 2024-02-10T09:48:40.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:39.308563+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e85: 8 total, 7 up, 8 in 2024-02-10T09:48:40.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:40 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.2.service: Succeeded. 2024-02-10T09:48:40.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:40 smithi153 systemd[1]: Stopped Ceph osd.2 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:48:40.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:40 smithi153 systemd[1]: Starting Ceph osd.2 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:48:41.657 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:39.945196+0000 mgr.y (mgr.25138) 179 : cluster [DBG] pgmap v91: 129 pgs: 6 peering, 8 stale+active+clean, 115 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:41.658 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:39.945196+0000 mgr.y (mgr.25138) 179 : cluster [DBG] pgmap v91: 129 pgs: 6 peering, 8 stale+active+clean, 115 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:41.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:48:41.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T09:48:41.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-02-10T09:48:41.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T09:48:41.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-02-10T09:48:41.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T09:48:41.659 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T09:48:41.659 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:41 smithi153 bash[81700]: --> ceph-volume lvm activate successful for osd ID: 2 2024-02-10T09:48:42.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:42.324143+0000 mon.a (mon.0) 544 : cluster [WRN] Health check failed: Degraded data redundancy: 5/606 objects degraded (0.825%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:48:42.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:42.324143+0000 mon.a (mon.0) 544 : cluster [WRN] Health check failed: Degraded data redundancy: 5/606 objects degraded (0.825%), 1 pg degraded (PG_DEGRADED) 2024-02-10T09:48:42.439 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:48:42.276+0000 7fa1a2e54700 1 -- 172.21.15.153:0/1584385251 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556aa65fa680 con 0x556aa65c1000 2024-02-10T09:48:42.440 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:42 smithi153 bash[81945]: a3f67a37aa3017dfcfabe31bbf065afd452524340faad71880b99621fd01a516 2024-02-10T09:48:42.440 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:42 smithi153 systemd[1]: Started Ceph osd.2 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:48:43.189 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:48:42.802+0000 7fa1aa8e2380 -1 Falling back to public interface 2024-02-10T09:48:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:41.945780+0000 mgr.y (mgr.25138) 180 : cluster [DBG] pgmap v92: 129 pgs: 1 active+undersized+degraded, 6 peering, 6 stale+active+clean, 4 active+undersized, 112 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail; 5/606 objects degraded (0.825%) 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:42.410284+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:42.413918+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:42.414422+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:42.415783+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:41.945780+0000 mgr.y (mgr.25138) 180 : cluster [DBG] pgmap v92: 129 pgs: 1 active+undersized+degraded, 6 peering, 6 stale+active+clean, 4 active+undersized, 112 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail; 5/606 objects degraded (0.825%) 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:42.410284+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:42.413918+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:42.414422+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:43.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:42.415783+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:48:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:48:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:48:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:48:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:44] "GET /metrics HTTP/1.1" 200 162252 "" "Prometheus/2.33.4" 2024-02-10T09:48:45.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:43.946371+0000 mgr.y (mgr.25138) 181 : cluster [DBG] pgmap v93: 129 pgs: 10 active+undersized+degraded, 6 peering, 16 active+undersized, 97 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail; 36/606 objects degraded (5.941%) 2024-02-10T09:48:45.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:48:45.551+0000 7fa1aa8e2380 -1 osd.2 83 log_to_monitors {default=true} 2024-02-10T09:48:45.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:43.946371+0000 mgr.y (mgr.25138) 181 : cluster [DBG] pgmap v93: 129 pgs: 10 active+undersized+degraded, 6 peering, 16 active+undersized, 97 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail; 36/606 objects degraded (5.941%) 2024-02-10T09:48:46.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:48:46.440+0000 7fa19adc7700 -1 osd.2 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:48:46.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:45.555700+0000 mon.c (mon.1) 10 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/3760791576,v1:172.21.15.153:6819/3760791576]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:45.555988+0000 mon.a (mon.0) 549 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.139990+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.143722+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.148640+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.150724+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.152168+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.153275+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.154376+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.155488+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.156834+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.160024+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.160611+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.163177+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:48:46.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.163738+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:45.555700+0000 mon.c (mon.1) 10 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/3760791576,v1:172.21.15.153:6819/3760791576]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:45.555988+0000 mon.a (mon.0) 549 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.139990+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.143722+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.148640+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.150724+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.152168+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.153275+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.154376+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.155488+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.156834+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.160024+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.160611+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.163177+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.163738+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.166802+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:48:46.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.167374+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.167866+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.168311+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.168754+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.169199+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.169634+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.170088+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.170533+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.170956+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.171377+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.171788+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.172203+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.175134+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.175918+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.177860+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:48:46.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.178259+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.183521+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.187278+0000 mon.a (mon.0) 581 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.191347+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.166802+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.167374+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.167866+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.168311+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.168754+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.169199+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.169634+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.170088+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.170533+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.170956+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.171377+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.171788+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.172203+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.175134+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.175918+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.177860+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.178259+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.183521+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.187278+0000 mon.a (mon.0) 581 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:48:46.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:46 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.191347+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:45.946809+0000 mgr.y (mgr.25138) 182 : cluster [DBG] pgmap v94: 129 pgs: 12 active+undersized+degraded, 20 active+undersized, 97 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail; 54/606 objects degraded (8.911%) 2024-02-10T09:48:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:46.156079+0000 mgr.y (mgr.25138) 183 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:48:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:48:46.175590+0000 mgr.y (mgr.25138) 184 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:48:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.429360+0000 mon.a (mon.0) 583 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T09:48:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:46.429426+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e86: 8 total, 7 up, 8 in 2024-02-10T09:48:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.431371+0000 mon.c (mon.1) 11 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/3760791576,v1:172.21.15.153:6819/3760791576]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:46.431701+0000 mon.a (mon.0) 585 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:45.946809+0000 mgr.y (mgr.25138) 182 : cluster [DBG] pgmap v94: 129 pgs: 12 active+undersized+degraded, 20 active+undersized, 97 active+clean; 7.0 KiB data, 588 MiB used, 709 GiB / 715 GiB avail; 54/606 objects degraded (8.911%) 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:46.156079+0000 mgr.y (mgr.25138) 183 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:48:46.175590+0000 mgr.y (mgr.25138) 184 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.429360+0000 mon.a (mon.0) 583 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:46.429426+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e86: 8 total, 7 up, 8 in 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.431371+0000 mon.c (mon.1) 11 : audit [INF] from='osd.2 [v2:172.21.15.153:6818/3760791576,v1:172.21.15.153:6819/3760791576]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:48:47.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:46.431701+0000 mon.a (mon.0) 585 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:48:48.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:47.422398+0000 mon.a (mon.0) 586 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:48:48.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:47.427311+0000 mon.a (mon.0) 587 : cluster [INF] osd.2 [v2:172.21.15.153:6818/3760791576,v1:172.21.15.153:6819/3760791576] boot 2024-02-10T09:48:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:47.427369+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-10T09:48:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:47.428094+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:48:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:48.157786+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:48.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:48.429265+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-10T09:48:48.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:47.422398+0000 mon.a (mon.0) 586 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:48:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:47.427311+0000 mon.a (mon.0) 587 : cluster [INF] osd.2 [v2:172.21.15.153:6818/3760791576,v1:172.21.15.153:6819/3760791576] boot 2024-02-10T09:48:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:47.427369+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-10T09:48:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:47.428094+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T09:48:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:48.157786+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:48.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:48.429265+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-10T09:48:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:47.947374+0000 mgr.y (mgr.25138) 185 : cluster [DBG] pgmap v97: 129 pgs: 12 active+undersized+degraded, 20 active+undersized, 97 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 54/606 objects degraded (8.911%) 2024-02-10T09:48:49.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:47.947374+0000 mgr.y (mgr.25138) 185 : cluster [DBG] pgmap v97: 129 pgs: 12 active+undersized+degraded, 20 active+undersized, 97 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 54/606 objects degraded (8.911%) 2024-02-10T09:48:50.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:50.093613+0000 mon.a (mon.0) 592 : cluster [WRN] Health check update: Degraded data redundancy: 54/606 objects degraded (8.911%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T09:48:50.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:50.093613+0000 mon.a (mon.0) 592 : cluster [WRN] Health check update: Degraded data redundancy: 54/606 objects degraded (8.911%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T09:48:51.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:49.948021+0000 mgr.y (mgr.25138) 186 : cluster [DBG] pgmap v99: 129 pgs: 15 peering, 2 active+undersized+degraded, 11 active+undersized, 101 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 6/606 objects degraded (0.990%) 2024-02-10T09:48:51.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:49.948021+0000 mgr.y (mgr.25138) 186 : cluster [DBG] pgmap v99: 129 pgs: 15 peering, 2 active+undersized+degraded, 11 active+undersized, 101 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 6/606 objects degraded (0.990%) 2024-02-10T09:48:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:52.460820+0000 mon.a (mon.0) 593 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/606 objects degraded (0.990%), 2 pgs degraded) 2024-02-10T09:48:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:52.460854+0000 mon.a (mon.0) 594 : cluster [INF] Cluster is now healthy 2024-02-10T09:48:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:52.460820+0000 mon.a (mon.0) 593 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/606 objects degraded (0.990%), 2 pgs degraded) 2024-02-10T09:48:52.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:52.460854+0000 mon.a (mon.0) 594 : cluster [INF] Cluster is now healthy 2024-02-10T09:48:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:51.948571+0000 mgr.y (mgr.25138) 187 : cluster [DBG] pgmap v100: 129 pgs: 15 peering, 114 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:48:53.167400+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:53.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:51.948571+0000 mgr.y (mgr.25138) 187 : cluster [DBG] pgmap v100: 129 pgs: 15 peering, 114 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:48:53.167400+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:48:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:48:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:48:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:48:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:48:54] "GET /metrics HTTP/1.1" 200 163033 "" "Prometheus/2.33.4" 2024-02-10T09:48:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:53.949246+0000 mgr.y (mgr.25138) 188 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:55.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:53.949246+0000 mgr.y (mgr.25138) 188 : cluster [DBG] pgmap v101: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:57.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:55.949766+0000 mgr.y (mgr.25138) 189 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:57.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:55.949766+0000 mgr.y (mgr.25138) 189 : cluster [DBG] pgmap v102: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:48:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:57.950382+0000 mgr.y (mgr.25138) 190 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:48:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:48:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:57.950382+0000 mgr.y (mgr.25138) 190 : cluster [DBG] pgmap v103: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:01.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:48:59.951279+0000 mgr.y (mgr.25138) 191 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:01.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:48:59.951279+0000 mgr.y (mgr.25138) 191 : cluster [DBG] pgmap v104: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:01.951805+0000 mgr.y (mgr.25138) 192 : cluster [DBG] pgmap v105: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:03.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:01.951805+0000 mgr.y (mgr.25138) 192 : cluster [DBG] pgmap v105: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:49:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:49:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:04] "GET /metrics HTTP/1.1" 200 163033 "" "Prometheus/2.33.4" 2024-02-10T09:49:05.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:03.952305+0000 mgr.y (mgr.25138) 193 : cluster [DBG] pgmap v106: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:05.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:03.952305+0000 mgr.y (mgr.25138) 193 : cluster [DBG] pgmap v106: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:06.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:49:06.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:05.333061+0000 mgr.y (mgr.25138) 194 : audit [DBG] from='client.64282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:06.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:05.333061+0000 mgr.y (mgr.25138) 194 : audit [DBG] from='client.64282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:07.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:05.952899+0000 mgr.y (mgr.25138) 195 : cluster [DBG] pgmap v107: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T09:49:07.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:05.952899+0000 mgr.y (mgr.25138) 195 : cluster [DBG] pgmap v107: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (17m) 21s ago 17m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (4m) 2m ago 20m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (5m) 2m ago 30m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (13m) 21s ago 32m 432M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (3m) 21s ago 32m 69.8M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (3m) 2m ago 30m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (2m) 21s ago 31m 63.0M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (5m) 21s ago 22m 21.2M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (5m) 2m ago 22m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (105s) 21s ago 30m 39.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:49:07.940 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (77s) 21s ago 29m 38.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (25s) 21s ago 29m 13.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (29m) 21s ago 29m 49.5M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (28m) 2m ago 28m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (28m) 2m ago 28m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (28m) 2m ago 28m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (28m) 2m ago 28m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (17m) 2m ago 17m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (20m) 21s ago 20m 37.7M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:49:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (20m) 2m ago 20m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:49:08.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.928610+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.929699+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.930018+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.930481+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.930683+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.930890+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.931082+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.931249+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.931443+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.931777+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.933325+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:08.895 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.933571+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.933829+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.934068+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.934302+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.928610+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.929699+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.930018+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.930481+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.930683+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.930890+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.931082+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.931249+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.931443+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.931777+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.933325+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.933571+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.933829+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.934068+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:08.897 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.934302+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:09.685 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:07.926740+0000 mgr.y (mgr.25138) 196 : audit [DBG] from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:09.685 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:07.953288+0000 mgr.y (mgr.25138) 197 : cluster [DBG] pgmap v108: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 27 op/s 2024-02-10T09:49:09.686 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:07.926740+0000 mgr.y (mgr.25138) 196 : audit [DBG] from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:09.686 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:07.953288+0000 mgr.y (mgr.25138) 197 : cluster [DBG] pgmap v108: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 27 op/s 2024-02-10T09:49:10.431 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:49:10.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-02-10T09:49:11.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:09.953968+0000 mgr.y (mgr.25138) 198 : cluster [DBG] pgmap v109: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:11.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:10.419604+0000 mon.c (mon.1) 12 : audit [DBG] from='client.54342 172.21.15.153:0/1185329795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:11.375 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:09.953968+0000 mgr.y (mgr.25138) 198 : cluster [DBG] pgmap v109: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:11.376 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:10.419604+0000 mon.c (mon.1) 12 : audit [DBG] from='client.54342 172.21.15.153:0/1185329795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:13.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:11.954541+0000 mgr.y (mgr.25138) 199 : cluster [DBG] pgmap v110: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:13.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:11.954541+0000 mgr.y (mgr.25138) 199 : cluster [DBG] pgmap v110: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:49:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:49:14.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:12.952654+0000 mgr.y (mgr.25138) 200 : audit [DBG] from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:14.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:12.952654+0000 mgr.y (mgr.25138) 200 : audit [DBG] from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:14.538 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:14] "GET /metrics HTTP/1.1" 200 163212 "" "Prometheus/2.33.4" 2024-02-10T09:49:14.773 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:49:15.382 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:13.955260+0000 mgr.y (mgr.25138) 201 : cluster [DBG] pgmap v111: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:15.382 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:13.955260+0000 mgr.y (mgr.25138) 201 : cluster [DBG] pgmap v111: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:15.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-02-10T09:49:17.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:15.955778+0000 mgr.y (mgr.25138) 202 : cluster [DBG] pgmap v112: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:17.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:15.955778+0000 mgr.y (mgr.25138) 202 : cluster [DBG] pgmap v112: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T09:49:18.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:17.497194+0000 mgr.y (mgr.25138) 203 : audit [DBG] from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:18.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:17.497194+0000 mgr.y (mgr.25138) 203 : audit [DBG] from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:19.309 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:17.956274+0000 mgr.y (mgr.25138) 204 : cluster [DBG] pgmap v113: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s 2024-02-10T09:49:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:17.956274+0000 mgr.y (mgr.25138) 204 : cluster [DBG] pgmap v113: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s 2024-02-10T09:49:19.331 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:49:20.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:49:20.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:19.326555+0000 mgr.y (mgr.25138) 205 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:49:20.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:19.330685+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:20.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:19.334993+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:20.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:19.340337+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:20.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:19.344034+0000 mgr.y (mgr.25138) 206 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:49:20.646 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:19.326555+0000 mgr.y (mgr.25138) 205 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:49:20.646 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:19.330685+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:20.646 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:19.334993+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:20.646 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:19.340337+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:20.646 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:19.344034+0000 mgr.y (mgr.25138) 206 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:49:21.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:19.957056+0000 mgr.y (mgr.25138) 207 : cluster [DBG] pgmap v114: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 92 op/s 2024-02-10T09:49:21.570 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:19.957056+0000 mgr.y (mgr.25138) 207 : cluster [DBG] pgmap v114: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 92 op/s 2024-02-10T09:49:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.812542+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:21.812864+0000 mgr.y (mgr.25138) 208 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:21.812919+0000 mgr.y (mgr.25138) 209 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.813735+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.817884+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.819035+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:21.819591+0000 mgr.y (mgr.25138) 210 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.823114+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.824115+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:21.824616+0000 mgr.y (mgr.25138) 211 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.828493+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.829318+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:21.829815+0000 mgr.y (mgr.25138) 212 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.834310+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.835838+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:21.835990+0000 mgr.y (mgr.25138) 213 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:21.836867+0000 mgr.y (mgr.25138) 214 : cephadm [INF] Upgrade: osd.3 is safe to restart 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:21.957490+0000 mgr.y (mgr.25138) 215 : cluster [DBG] pgmap v115: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.812542+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:21.812864+0000 mgr.y (mgr.25138) 208 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:21.812919+0000 mgr.y (mgr.25138) 209 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.813735+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.817884+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.819035+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:21.819591+0000 mgr.y (mgr.25138) 210 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.823114+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.824115+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:21.824616+0000 mgr.y (mgr.25138) 211 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.828493+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.829318+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:21.829815+0000 mgr.y (mgr.25138) 212 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.834310+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.835838+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:21.835990+0000 mgr.y (mgr.25138) 213 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:21.836867+0000 mgr.y (mgr.25138) 214 : cephadm [INF] Upgrade: osd.3 is safe to restart 2024-02-10T09:49:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:21.957490+0000 mgr.y (mgr.25138) 215 : cluster [DBG] pgmap v115: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:23.248 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:49:23.899 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (17m) 37s ago 18m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (5m) 3m ago 20m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (5m) 3m ago 30m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (13m) 37s ago 32m 432M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (3m) 37s ago 32m 69.8M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (4m) 3m ago 31m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (3m) 37s ago 31m 63.0M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (5m) 37s ago 23m 21.2M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (5m) 3m ago 23m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (2m) 37s ago 30m 39.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:49:23.900 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (93s) 37s ago 30m 38.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (41s) 37s ago 29m 13.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (29m) 37s ago 29m 49.5M 4096M 15.2.0 204a01f9b0b6 90119323f9b5 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (29m) 3m ago 29m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (28m) 3m ago 28m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (28m) 3m ago 28m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (28m) 3m ago 28m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (17m) 3m ago 17m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (20m) 37s ago 20m 37.7M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:49:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (20m) 3m ago 20m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:49:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:49:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:49:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.236888+0000 mgr.y (mgr.25138) 216 : audit [DBG] from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.557757+0000 mgr.y (mgr.25138) 217 : audit [DBG] from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.790707+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.791421+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:49:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.791941+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:49:24.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.793315+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "container_image"}]: dispatch 2024-02-10T09:49:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.236888+0000 mgr.y (mgr.25138) 216 : audit [DBG] from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.557757+0000 mgr.y (mgr.25138) 217 : audit [DBG] from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.790707+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.791421+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T09:49:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.791941+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:49:24.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.793315+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "container_image"}]: dispatch 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:49:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 5, 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 7, 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 8 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:49:24.295 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:49:24.618 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:49:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:49:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:49:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T09:49:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:49:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "3/8 daemons upgraded", 2024-02-10T09:49:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:49:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:49:24.619 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:49:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:24] "GET /metrics HTTP/1.1" 200 163223 "" "Prometheus/2.33.4" 2024-02-10T09:49:25.084 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:23.785214+0000 mgr.y (mgr.25138) 218 : cephadm [INF] Upgrade: Updating osd.3 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:23.792983+0000 mgr.y (mgr.25138) 219 : cephadm [INF] Deploying daemon osd.3 on smithi153 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.887096+0000 mgr.y (mgr.25138) 220 : audit [DBG] from='client.64324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.888967+0000 mon.a (mon.0) 628 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.889376+0000 mon.a (mon.0) 629 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.889698+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.890070+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.890390+0000 mon.a (mon.0) 632 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.890706+0000 mon.a (mon.0) 633 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.892327+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.892676+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.893011+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.893356+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:23.893706+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.085 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:23.958208+0000 mgr.y (mgr.25138) 221 : cluster [DBG] pgmap v116: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:24.293625+0000 mon.c (mon.1) 13 : audit [DBG] from='client.64330 172.21.15.153:0/503776008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:23.785214+0000 mgr.y (mgr.25138) 218 : cephadm [INF] Upgrade: Updating osd.3 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:23.792983+0000 mgr.y (mgr.25138) 219 : cephadm [INF] Deploying daemon osd.3 on smithi153 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.887096+0000 mgr.y (mgr.25138) 220 : audit [DBG] from='client.64324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.888967+0000 mon.a (mon.0) 628 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.889376+0000 mon.a (mon.0) 629 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.889698+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.890070+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.086 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.890390+0000 mon.a (mon.0) 632 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.890706+0000 mon.a (mon.0) 633 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.892327+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.892676+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.893011+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.893356+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:23.893706+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:23.958208+0000 mgr.y (mgr.25138) 221 : cluster [DBG] pgmap v116: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:25.087 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:24.293625+0000 mon.c (mon.1) 13 : audit [DBG] from='client.64330 172.21.15.153:0/503776008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:26.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:24.617934+0000 mgr.y (mgr.25138) 222 : audit [DBG] from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:26.319 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:24.617934+0000 mgr.y (mgr.25138) 222 : audit [DBG] from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:26.587 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:26 smithi153 systemd[1]: Stopping Ceph osd.3 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:49:26.587 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:26 smithi153 bash[47529]: debug 2024-02-10T09:49:26.540+0000 7f9c04803700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:49:26.587 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:26 smithi153 bash[47529]: debug 2024-02-10T09:49:26.540+0000 7f9c04803700 -1 osd.3 88 *** Got signal Terminated *** 2024-02-10T09:49:26.587 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:26 smithi153 bash[47529]: debug 2024-02-10T09:49:26.540+0000 7f9c04803700 -1 osd.3 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:49:27.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:26 smithi153 bash[84268]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.3 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:25.958840+0000 mgr.y (mgr.25138) 223 : cluster [DBG] pgmap v117: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784068+0000 mon.a (mon.0) 639 : cluster [DBG] osd.3 reported immediately failed by osd.6 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784107+0000 mon.a (mon.0) 640 : cluster [INF] osd.3 failed (root=default,host=smithi153) (connection refused reported by osd.6) 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784279+0000 mon.a (mon.0) 641 : cluster [DBG] osd.3 reported immediately failed by osd.7 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784415+0000 mon.a (mon.0) 642 : cluster [DBG] osd.3 reported immediately failed by osd.0 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784517+0000 mon.a (mon.0) 643 : cluster [DBG] osd.3 reported immediately failed by osd.4 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784644+0000 mon.a (mon.0) 644 : cluster [DBG] osd.3 reported immediately failed by osd.1 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784770+0000 mon.a (mon.0) 645 : cluster [DBG] osd.3 reported immediately failed by osd.5 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.784881+0000 mon.a (mon.0) 646 : cluster [DBG] osd.3 reported immediately failed by osd.2 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785008+0000 mon.a (mon.0) 647 : cluster [DBG] osd.3 reported immediately failed by osd.6 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785137+0000 mon.a (mon.0) 648 : cluster [DBG] osd.3 reported immediately failed by osd.0 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785235+0000 mon.a (mon.0) 649 : cluster [DBG] osd.3 reported immediately failed by osd.7 2024-02-10T09:49:27.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785362+0000 mon.a (mon.0) 650 : cluster [DBG] osd.3 reported immediately failed by osd.1 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785458+0000 mon.a (mon.0) 651 : cluster [DBG] osd.3 reported immediately failed by osd.4 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785587+0000 mon.a (mon.0) 652 : cluster [DBG] osd.3 reported immediately failed by osd.2 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:26.785687+0000 mon.a (mon.0) 653 : cluster [DBG] osd.3 reported immediately failed by osd.5 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:25.958840+0000 mgr.y (mgr.25138) 223 : cluster [DBG] pgmap v117: 129 pgs: 129 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784068+0000 mon.a (mon.0) 639 : cluster [DBG] osd.3 reported immediately failed by osd.6 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784107+0000 mon.a (mon.0) 640 : cluster [INF] osd.3 failed (root=default,host=smithi153) (connection refused reported by osd.6) 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784279+0000 mon.a (mon.0) 641 : cluster [DBG] osd.3 reported immediately failed by osd.7 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784415+0000 mon.a (mon.0) 642 : cluster [DBG] osd.3 reported immediately failed by osd.0 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784517+0000 mon.a (mon.0) 643 : cluster [DBG] osd.3 reported immediately failed by osd.4 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784644+0000 mon.a (mon.0) 644 : cluster [DBG] osd.3 reported immediately failed by osd.1 2024-02-10T09:49:27.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784770+0000 mon.a (mon.0) 645 : cluster [DBG] osd.3 reported immediately failed by osd.5 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.784881+0000 mon.a (mon.0) 646 : cluster [DBG] osd.3 reported immediately failed by osd.2 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785008+0000 mon.a (mon.0) 647 : cluster [DBG] osd.3 reported immediately failed by osd.6 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785137+0000 mon.a (mon.0) 648 : cluster [DBG] osd.3 reported immediately failed by osd.0 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785235+0000 mon.a (mon.0) 649 : cluster [DBG] osd.3 reported immediately failed by osd.7 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785362+0000 mon.a (mon.0) 650 : cluster [DBG] osd.3 reported immediately failed by osd.1 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785458+0000 mon.a (mon.0) 651 : cluster [DBG] osd.3 reported immediately failed by osd.4 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785587+0000 mon.a (mon.0) 652 : cluster [DBG] osd.3 reported immediately failed by osd.2 2024-02-10T09:49:27.192 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:26.785687+0000 mon.a (mon.0) 653 : cluster [DBG] osd.3 reported immediately failed by osd.5 2024-02-10T09:49:28.386 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:27.110710+0000 mon.a (mon.0) 654 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:49:28.386 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:27.121325+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e89: 8 total, 7 up, 8 in 2024-02-10T09:49:28.387 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:27.110710+0000 mon.a (mon.0) 654 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:49:28.387 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:27.121325+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e89: 8 total, 7 up, 8 in 2024-02-10T09:49:29.426 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:27.959260+0000 mgr.y (mgr.25138) 224 : cluster [DBG] pgmap v119: 129 pgs: 20 stale+active+clean, 109 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:29.426 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:28.121282+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e90: 8 total, 7 up, 8 in 2024-02-10T09:49:29.427 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:28 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.3.service: Succeeded. 2024-02-10T09:49:29.427 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:28 smithi153 systemd[1]: Stopped Ceph osd.3 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:49:29.427 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 systemd[1]: Starting Ceph osd.3 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:49:29.427 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:27.959260+0000 mgr.y (mgr.25138) 224 : cluster [DBG] pgmap v119: 129 pgs: 20 stale+active+clean, 109 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail 2024-02-10T09:49:29.427 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:28.121282+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e90: 8 total, 7 up, 8 in 2024-02-10T09:49:30.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:49:30.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T09:49:30.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-02-10T09:49:30.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T09:49:30.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-02-10T09:49:30.190 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T09:49:30.190 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T09:49:30.190 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:29 smithi153 bash[84567]: --> ceph-volume lvm activate successful for osd ID: 3 2024-02-10T09:49:31.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:49:31.069+0000 7f2a51d18700 1 -- 172.21.15.153:0/1553578182 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dbfda9a680 con 0x55dbfda61800 2024-02-10T09:49:31.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:31 smithi153 bash[84811]: 1200374c128b222c3ac51ba332192275591ecc811728756fe4b9480a151de9c9 2024-02-10T09:49:31.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:31 smithi153 systemd[1]: Started Ceph osd.3 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:49:31.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:29.959865+0000 mgr.y (mgr.25138) 225 : cluster [DBG] pgmap v121: 129 pgs: 5 active+undersized+degraded, 22 peering, 5 stale+active+clean, 8 active+undersized, 89 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 18/609 objects degraded (2.956%) 2024-02-10T09:49:31.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:30.120398+0000 mon.a (mon.0) 657 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T09:49:31.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:30.120420+0000 mon.a (mon.0) 658 : cluster [WRN] Health check failed: Degraded data redundancy: 18/609 objects degraded (2.956%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T09:49:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:29.959865+0000 mgr.y (mgr.25138) 225 : cluster [DBG] pgmap v121: 129 pgs: 5 active+undersized+degraded, 22 peering, 5 stale+active+clean, 8 active+undersized, 89 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 18/609 objects degraded (2.956%) 2024-02-10T09:49:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:30.120398+0000 mon.a (mon.0) 657 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T09:49:31.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:30.120420+0000 mon.a (mon.0) 658 : cluster [WRN] Health check failed: Degraded data redundancy: 18/609 objects degraded (2.956%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T09:49:31.939 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:49:31.584+0000 7f2a597a6380 -1 Falling back to public interface 2024-02-10T09:49:32.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:31.200304+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:32.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:31.201109+0000 mon.a (mon.0) 660 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:32.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:31.202347+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:49:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:31.200304+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:31.201109+0000 mon.a (mon.0) 660 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:31.202347+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:49:33.596 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:31.960472+0000 mgr.y (mgr.25138) 226 : cluster [DBG] pgmap v122: 129 pgs: 12 active+undersized+degraded, 22 peering, 16 active+undersized, 79 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 42/609 objects degraded (6.897%) 2024-02-10T09:49:33.596 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:31.960472+0000 mgr.y (mgr.25138) 226 : cluster [DBG] pgmap v122: 129 pgs: 12 active+undersized+degraded, 22 peering, 16 active+undersized, 79 active+clean; 7.0 KiB data, 878 MiB used, 709 GiB / 715 GiB avail; 42/609 objects degraded (6.897%) 2024-02-10T09:49:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:49:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:49:34.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:34.204590+0000 mon.a (mon.0) 662 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T09:49:34.413 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:49:34.338+0000 7f2a597a6380 -1 osd.3 88 log_to_monitors {default=true} 2024-02-10T09:49:34.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:34.204590+0000 mon.a (mon.0) 662 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T09:49:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:34] "GET /metrics HTTP/1.1" 200 163223 "" "Prometheus/2.33.4" 2024-02-10T09:49:35.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:49:35.280+0000 7f2a49c8b700 -1 osd.3 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:49:35.057+0000 7fc4ea3a1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:33.961210+0000 mgr.y (mgr.25138) 227 : cluster [DBG] pgmap v123: 129 pgs: 22 active+undersized+degraded, 28 active+undersized, 79 active+clean; 7.0 KiB data, 879 MiB used, 709 GiB / 715 GiB avail; 77/609 objects degraded (12.644%) 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:34.343428+0000 mon.a (mon.0) 663 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:34.773951+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.049783+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.054423+0000 mon.a (mon.0) 666 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.056252+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.057302+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:35.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.058158+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.059247+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:33.961210+0000 mgr.y (mgr.25138) 227 : cluster [DBG] pgmap v123: 129 pgs: 22 active+undersized+degraded, 28 active+undersized, 79 active+clean; 7.0 KiB data, 879 MiB used, 709 GiB / 715 GiB avail; 77/609 objects degraded (12.644%) 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:34.343428+0000 mon.a (mon.0) 663 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:34.773951+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.049783+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.054423+0000 mon.a (mon.0) 666 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.056252+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.057302+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.058158+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:35.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.059247+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.059367+0000 mgr.y (mgr.25138) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:35.059962+0000 mgr.y (mgr.25138) 229 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-10T09:49:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.265543+0000 mon.a (mon.0) 671 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T09:49:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:35.265607+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e91: 8 total, 7 up, 8 in 2024-02-10T09:49:36.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:35.266486+0000 mon.a (mon.0) 673 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:49:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.059367+0000 mgr.y (mgr.25138) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:35.059962+0000 mgr.y (mgr.25138) 229 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-10T09:49:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.265543+0000 mon.a (mon.0) 671 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T09:49:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:35.265607+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e91: 8 total, 7 up, 8 in 2024-02-10T09:49:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:35.266486+0000 mon.a (mon.0) 673 : audit [INF] from='osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-02-10T09:49:37.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:35.961636+0000 mgr.y (mgr.25138) 230 : cluster [DBG] pgmap v125: 129 pgs: 22 active+undersized+degraded, 28 active+undersized, 79 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 77/609 objects degraded (12.644%) 2024-02-10T09:49:37.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:36.269477+0000 mon.a (mon.0) 674 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:36.276751+0000 mon.a (mon.0) 675 : cluster [INF] osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323] boot 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:36.276804+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:36.279255+0000 mon.a (mon.0) 677 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:35.961636+0000 mgr.y (mgr.25138) 230 : cluster [DBG] pgmap v125: 129 pgs: 22 active+undersized+degraded, 28 active+undersized, 79 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 77/609 objects degraded (12.644%) 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:36.269477+0000 mon.a (mon.0) 674 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:36.276751+0000 mon.a (mon.0) 675 : cluster [INF] osd.3 [v2:172.21.15.153:6826/545618323,v1:172.21.15.153:6827/545618323] boot 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:36.276804+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-10T09:49:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:36.279255+0000 mon.a (mon.0) 677 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T09:49:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:37.289095+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-10T09:49:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:38.078839+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:38.188270+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:38.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:37.289095+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-10T09:49:38.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:38.078839+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:38.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:38.188270+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:39.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:37.962059+0000 mgr.y (mgr.25138) 231 : cluster [DBG] pgmap v128: 129 pgs: 10 peering, 18 active+undersized+degraded, 22 active+undersized, 79 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 62/609 objects degraded (10.181%) 2024-02-10T09:49:39.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:38.283781+0000 mon.a (mon.0) 681 : cluster [WRN] Health check update: Degraded data redundancy: 62/609 objects degraded (10.181%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T09:49:39.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:37.962059+0000 mgr.y (mgr.25138) 231 : cluster [DBG] pgmap v128: 129 pgs: 10 peering, 18 active+undersized+degraded, 22 active+undersized, 79 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 62/609 objects degraded (10.181%) 2024-02-10T09:49:39.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:38.283781+0000 mon.a (mon.0) 681 : cluster [WRN] Health check update: Degraded data redundancy: 62/609 objects degraded (10.181%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T09:49:41.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:39.962742+0000 mgr.y (mgr.25138) 232 : cluster [DBG] pgmap v129: 129 pgs: 10 peering, 9 active+undersized+degraded, 11 active+undersized, 99 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 33/609 objects degraded (5.419%) 2024-02-10T09:49:41.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:39.962742+0000 mgr.y (mgr.25138) 232 : cluster [DBG] pgmap v129: 129 pgs: 10 peering, 9 active+undersized+degraded, 11 active+undersized, 99 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 33/609 objects degraded (5.419%) 2024-02-10T09:49:42.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:42.295846+0000 mon.a (mon.0) 682 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/609 objects degraded (5.419%), 9 pgs degraded) 2024-02-10T09:49:42.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:42.295908+0000 mon.a (mon.0) 683 : cluster [INF] Cluster is now healthy 2024-02-10T09:49:42.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:42.295846+0000 mon.a (mon.0) 682 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/609 objects degraded (5.419%), 9 pgs degraded) 2024-02-10T09:49:42.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:42 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:42.295908+0000 mon.a (mon.0) 683 : cluster [INF] Cluster is now healthy 2024-02-10T09:49:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:41.963370+0000 mgr.y (mgr.25138) 233 : cluster [DBG] pgmap v130: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:41.963370+0000 mgr.y (mgr.25138) 233 : cluster [DBG] pgmap v130: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:49:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:49:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:44] "GET /metrics HTTP/1.1" 200 163790 "" "Prometheus/2.33.4" 2024-02-10T09:49:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:43.963972+0000 mgr.y (mgr.25138) 234 : cluster [DBG] pgmap v131: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:45.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:43.963972+0000 mgr.y (mgr.25138) 234 : cluster [DBG] pgmap v131: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:45.964554+0000 mgr.y (mgr.25138) 235 : cluster [DBG] pgmap v132: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:45.964554+0000 mgr.y (mgr.25138) 235 : cluster [DBG] pgmap v132: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:49.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:47.965223+0000 mgr.y (mgr.25138) 236 : cluster [DBG] pgmap v133: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:49.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:48.201702+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:49.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:47.965223+0000 mgr.y (mgr.25138) 236 : cluster [DBG] pgmap v133: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:49.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:48.201702+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:50.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:50.072222+0000 mon.a (mon.0) 685 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:50.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:50.072222+0000 mon.a (mon.0) 685 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:51.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:49.965880+0000 mgr.y (mgr.25138) 237 : cluster [DBG] pgmap v134: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:51.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:50.072405+0000 mgr.y (mgr.25138) 238 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:51.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:50.073484+0000 mgr.y (mgr.25138) 239 : cephadm [INF] Upgrade: osd.4 is safe to restart 2024-02-10T09:49:51.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:49.965880+0000 mgr.y (mgr.25138) 237 : cluster [DBG] pgmap v134: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:51.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:50.072405+0000 mgr.y (mgr.25138) 238 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T09:49:51.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:50.073484+0000 mgr.y (mgr.25138) 239 : cephadm [INF] Upgrade: osd.4 is safe to restart 2024-02-10T09:49:52.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:51.485106+0000 mgr.y (mgr.25138) 240 : cephadm [INF] Upgrade: Updating osd.4 2024-02-10T09:49:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:51.491172+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:51.492050+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:49:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:51.492763+0000 mon.a (mon.0) 688 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:49:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:49:51.494170+0000 mgr.y (mgr.25138) 241 : cephadm [INF] Deploying daemon osd.4 on smithi204 2024-02-10T09:49:52.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:51.494615+0000 mon.a (mon.0) 689 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "container_image"}]: dispatch 2024-02-10T09:49:52.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:51.485106+0000 mgr.y (mgr.25138) 240 : cephadm [INF] Upgrade: Updating osd.4 2024-02-10T09:49:52.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:51.491172+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:52.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:51.492050+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T09:49:52.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:51.492763+0000 mon.a (mon.0) 688 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:49:52.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:49:51.494170+0000 mgr.y (mgr.25138) 241 : cephadm [INF] Deploying daemon osd.4 on smithi204 2024-02-10T09:49:52.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:51.494615+0000 mon.a (mon.0) 689 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "container_image"}]: dispatch 2024-02-10T09:49:53.831 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:49:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:49:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:51.966411+0000 mgr.y (mgr.25138) 242 : cluster [DBG] pgmap v135: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:53.078658+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:53.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:51.966411+0000 mgr.y (mgr.25138) 242 : cluster [DBG] pgmap v135: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:53.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:53.078658+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:54.446 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:54 smithi204 systemd[1]: Stopping Ceph osd.4 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:49:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:49:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:49:54] "GET /metrics HTTP/1.1" 200 164023 "" "Prometheus/2.33.4" 2024-02-10T09:49:54.831 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:54 smithi204 bash[38176]: debug 2024-02-10T09:49:54.445+0000 7f77955e9700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:49:54.832 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:54 smithi204 bash[38176]: debug 2024-02-10T09:49:54.445+0000 7f77955e9700 -1 osd.4 93 *** Got signal Terminated *** 2024-02-10T09:49:54.832 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:54 smithi204 bash[38176]: debug 2024-02-10T09:49:54.445+0000 7f77955e9700 -1 osd.4 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:49:54.956 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:49:55.113 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:54 smithi204 bash[62693]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.4 2024-02-10T09:49:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:49:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (17m) 20s ago 18m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:49:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (5m) 3m ago 20m 49.3M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:49:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (6m) 3m ago 31m 383M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:49:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (14m) 20s ago 33m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:49:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (4m) 20s ago 33m 84.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (4m) 3m ago 31m 61.3M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (3m) 20s ago 31m 69.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (6m) 20s ago 23m 21.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (5m) 3m ago 23m 14.3M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (2m) 20s ago 31m 40.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 20s ago 30m 40.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (73s) 20s ago 30m 40.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (24s) 20s ago 30m 10.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (29m) 3m ago 29m 47.9M 4096M 15.2.0 204a01f9b0b6 d707da0e0933 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (29m) 3m ago 29m 47.8M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (29m) 3m ago 29m 47.1M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (28m) 3m ago 28m 45.9M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (18m) 3m ago 18m 74.5M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (20m) 20s ago 20m 39.1M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:49:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (20m) 3m ago 20m 38.0M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:49:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:53.967148+0000 mgr.y (mgr.25138) 243 : cluster [DBG] pgmap v136: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.689315+0000 mon.a (mon.0) 691 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.689359+0000 mon.a (mon.0) 692 : cluster [INF] osd.4 failed (root=default,host=smithi204) (connection refused reported by osd.3) 2024-02-10T09:49:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.689645+0000 mon.a (mon.0) 693 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.689898+0000 mon.a (mon.0) 694 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690042+0000 mon.a (mon.0) 695 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690161+0000 mon.a (mon.0) 696 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690294+0000 mon.a (mon.0) 697 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690411+0000 mon.a (mon.0) 698 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690558+0000 mon.a (mon.0) 699 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690676+0000 mon.a (mon.0) 700 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690818+0000 mon.a (mon.0) 701 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.690933+0000 mon.a (mon.0) 702 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.691067+0000 mon.a (mon.0) 703 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.691183+0000 mon.a (mon.0) 704 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.691317+0000 mon.a (mon.0) 705 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.691428+0000 mon.a (mon.0) 706 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:54.691534+0000 mon.a (mon.0) 707 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090108+0000 mon.a (mon.0) 708 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090180+0000 mon.a (mon.0) 709 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:53.967148+0000 mgr.y (mgr.25138) 243 : cluster [DBG] pgmap v136: 129 pgs: 129 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.689315+0000 mon.a (mon.0) 691 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.689359+0000 mon.a (mon.0) 692 : cluster [INF] osd.4 failed (root=default,host=smithi204) (connection refused reported by osd.3) 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.689645+0000 mon.a (mon.0) 693 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.689898+0000 mon.a (mon.0) 694 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690042+0000 mon.a (mon.0) 695 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690161+0000 mon.a (mon.0) 696 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690294+0000 mon.a (mon.0) 697 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690411+0000 mon.a (mon.0) 698 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690558+0000 mon.a (mon.0) 699 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690676+0000 mon.a (mon.0) 700 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690818+0000 mon.a (mon.0) 701 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.690933+0000 mon.a (mon.0) 702 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.691067+0000 mon.a (mon.0) 703 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.691183+0000 mon.a (mon.0) 704 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.691317+0000 mon.a (mon.0) 705 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.691428+0000 mon.a (mon.0) 706 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:54.691534+0000 mon.a (mon.0) 707 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090108+0000 mon.a (mon.0) 708 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090180+0000 mon.a (mon.0) 709 : cluster [DBG] osd.4 reported immediately failed by osd.3 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090385+0000 mon.a (mon.0) 710 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090541+0000 mon.a (mon.0) 711 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090665+0000 mon.a (mon.0) 712 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090794+0000 mon.a (mon.0) 713 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.090889+0000 mon.a (mon.0) 714 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091007+0000 mon.a (mon.0) 715 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091105+0000 mon.a (mon.0) 716 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091218+0000 mon.a (mon.0) 717 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091312+0000 mon.a (mon.0) 718 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091398+0000 mon.a (mon.0) 719 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091520+0000 mon.a (mon.0) 720 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091614+0000 mon.a (mon.0) 721 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091707+0000 mon.a (mon.0) 722 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.091811+0000 mon.a (mon.0) 723 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090385+0000 mon.a (mon.0) 710 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090541+0000 mon.a (mon.0) 711 : cluster [DBG] osd.4 reported immediately failed by osd.1 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090665+0000 mon.a (mon.0) 712 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090794+0000 mon.a (mon.0) 713 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.090889+0000 mon.a (mon.0) 714 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091007+0000 mon.a (mon.0) 715 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091105+0000 mon.a (mon.0) 716 : cluster [DBG] osd.4 reported immediately failed by osd.2 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091218+0000 mon.a (mon.0) 717 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091312+0000 mon.a (mon.0) 718 : cluster [DBG] osd.4 reported immediately failed by osd.0 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091398+0000 mon.a (mon.0) 719 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091520+0000 mon.a (mon.0) 720 : cluster [DBG] osd.4 reported immediately failed by osd.5 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091614+0000 mon.a (mon.0) 721 : cluster [DBG] osd.4 reported immediately failed by osd.6 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091707+0000 mon.a (mon.0) 722 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.091811+0000 mon.a (mon.0) 723 : cluster [DBG] osd.4 reported immediately failed by osd.7 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3, 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 4 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:55.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 5, 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 9 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:49:55.975 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:49:56.297 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "4/8 daemons upgraded", 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:49:56.298 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:49:56.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:54.943798+0000 mgr.y (mgr.25138) 244 : audit [DBG] from='client.54396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:56.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.261892+0000 mgr.y (mgr.25138) 245 : audit [DBG] from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:56.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.495933+0000 mon.a (mon.0) 724 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:49:56.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.501169+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e94: 8 total, 7 up, 8 in 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.578849+0000 mgr.y (mgr.25138) 246 : audit [DBG] from='client.64360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.580600+0000 mon.a (mon.0) 726 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.580973+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.581260+0000 mon.a (mon.0) 728 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.581567+0000 mon.a (mon.0) 729 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.581848+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.582138+0000 mon.a (mon.0) 731 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.583690+0000 mon.a (mon.0) 732 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.584004+0000 mon.a (mon.0) 733 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.584284+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:54.943798+0000 mgr.y (mgr.25138) 244 : audit [DBG] from='client.54396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.261892+0000 mgr.y (mgr.25138) 245 : audit [DBG] from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.495933+0000 mon.a (mon.0) 724 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.501169+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e94: 8 total, 7 up, 8 in 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.578849+0000 mgr.y (mgr.25138) 246 : audit [DBG] from='client.64360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.580600+0000 mon.a (mon.0) 726 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.580973+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.581260+0000 mon.a (mon.0) 728 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.581567+0000 mon.a (mon.0) 729 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.581848+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.582138+0000 mon.a (mon.0) 731 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.583690+0000 mon.a (mon.0) 732 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.584004+0000 mon.a (mon.0) 733 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.584284+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.584571+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.584844+0000 mon.a (mon.0) 736 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:55.973835+0000 mon.a (mon.0) 737 : audit [DBG] from='client.54414 172.21.15.153:0/1258056801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:56.501029+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e95: 8 total, 7 up, 8 in 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.584571+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.584844+0000 mon.a (mon.0) 736 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:55.973835+0000 mon.a (mon.0) 737 : audit [DBG] from='client.54414 172.21.15.153:0/1258056801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:49:56.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:56.501029+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e95: 8 total, 7 up, 8 in 2024-02-10T09:49:57.332 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:56 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.4.service: Succeeded. 2024-02-10T09:49:57.332 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:56 smithi204 systemd[1]: Stopped Ceph osd.4 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:49:57.332 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 systemd[1]: Starting Ceph osd.4 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:49:57.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:55.967484+0000 mgr.y (mgr.25138) 247 : cluster [DBG] pgmap v138: 129 pgs: 18 stale+active+clean, 111 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:57.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:56.297168+0000 mgr.y (mgr.25138) 248 : audit [DBG] from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:57.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:55.967484+0000 mgr.y (mgr.25138) 247 : cluster [DBG] pgmap v138: 129 pgs: 18 stale+active+clean, 111 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail 2024-02-10T09:49:57.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:56.297168+0000 mgr.y (mgr.25138) 248 : audit [DBG] from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T09:49:58.225 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T09:49:58.226 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:57 smithi204 bash[62996]: --> ceph-volume lvm activate successful for osd ID: 4 2024-02-10T09:49:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:58.522955+0000 mon.a (mon.0) 739 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:49:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:58.523031+0000 mon.a (mon.0) 740 : cluster [WRN] Health check failed: Degraded data redundancy: 24/609 objects degraded (3.941%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T09:49:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:58.522955+0000 mon.a (mon.0) 739 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T09:49:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:58.523031+0000 mon.a (mon.0) 740 : cluster [WRN] Health check failed: Degraded data redundancy: 24/609 objects degraded (3.941%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T09:49:59.080 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:49:58.957+0000 7fe25e29b700 1 -- 172.21.15.204:0/1742422797 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ba98c4680 con 0x564ba988b800 2024-02-10T09:49:59.080 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:59 smithi204 bash[63239]: 3b018a5578b9556d18003921762103f26ef11db7158e94b3190f4cb3984ce759 2024-02-10T09:49:59.332 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:59 smithi204 systemd[1]: Started Ceph osd.4 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:49:59.831 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:49:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:49:59.481+0000 7fe265d29380 -1 Falling back to public interface 2024-02-10T09:49:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:57.968023+0000 mgr.y (mgr.25138) 249 : cluster [DBG] pgmap v140: 129 pgs: 5 active+undersized+degraded, 3 peering, 14 stale+active+clean, 4 active+undersized, 103 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 24/609 objects degraded (3.941%) 2024-02-10T09:49:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:59.124893+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:59.125906+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:49:59.127674+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:49:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:57.968023+0000 mgr.y (mgr.25138) 249 : cluster [DBG] pgmap v140: 129 pgs: 5 active+undersized+degraded, 3 peering, 14 stale+active+clean, 4 active+undersized, 103 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 24/609 objects degraded (3.941%) 2024-02-10T09:49:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:59.124893+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:49:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:59.125906+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:49:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:49:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:49:59.127674+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000196+0000 mon.a (mon.0) 744 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 24/609 objects degraded (3.941%), 5 pgs degraded 2024-02-10T09:50:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000247+0000 mon.a (mon.0) 745 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-10T09:50:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000273+0000 mon.a (mon.0) 746 : cluster [WRN] osd.4 (root=default,host=smithi204) is down 2024-02-10T09:50:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000295+0000 mon.a (mon.0) 747 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000317+0000 mon.a (mon.0) 748 : cluster [WRN] pg 4.d is stuck peering for 71s, current state peering, last acting [2,1] 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000338+0000 mon.a (mon.0) 749 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 24/609 objects degraded (3.941%), 5 pgs degraded 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000363+0000 mon.a (mon.0) 750 : cluster [WRN] pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000401+0000 mon.a (mon.0) 751 : cluster [WRN] pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000420+0000 mon.a (mon.0) 752 : cluster [WRN] pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000434+0000 mon.a (mon.0) 753 : cluster [WRN] pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:00.000443+0000 mon.a (mon.0) 754 : cluster [WRN] pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000196+0000 mon.a (mon.0) 744 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 24/609 objects degraded (3.941%), 5 pgs degraded 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000247+0000 mon.a (mon.0) 745 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000273+0000 mon.a (mon.0) 746 : cluster [WRN] osd.4 (root=default,host=smithi204) is down 2024-02-10T09:50:00.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000295+0000 mon.a (mon.0) 747 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000317+0000 mon.a (mon.0) 748 : cluster [WRN] pg 4.d is stuck peering for 71s, current state peering, last acting [2,1] 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000338+0000 mon.a (mon.0) 749 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 24/609 objects degraded (3.941%), 5 pgs degraded 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000363+0000 mon.a (mon.0) 750 : cluster [WRN] pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000401+0000 mon.a (mon.0) 751 : cluster [WRN] pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000420+0000 mon.a (mon.0) 752 : cluster [WRN] pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000434+0000 mon.a (mon.0) 753 : cluster [WRN] pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-10T09:50:00.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:00.000443+0000 mon.a (mon.0) 754 : cluster [WRN] pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-10T09:50:01.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:49:59.968954+0000 mgr.y (mgr.25138) 250 : cluster [DBG] pgmap v141: 129 pgs: 19 active+undersized+degraded, 3 peering, 5 stale+active+clean, 26 active+undersized, 76 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 91/609 objects degraded (14.943%) 2024-02-10T09:50:01.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:49:59.968954+0000 mgr.y (mgr.25138) 250 : cluster [DBG] pgmap v141: 129 pgs: 19 active+undersized+degraded, 3 peering, 5 stale+active+clean, 26 active+undersized, 76 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 91/609 objects degraded (14.943%) 2024-02-10T09:50:02.582 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:50:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:50:02.239+0000 7fe265d29380 -1 osd.4 93 log_to_monitors {default=true} 2024-02-10T09:50:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:02.243938+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/320252880,v1:172.21.15.204:6801/320252880]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:50:02.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:02.244403+0000 mon.a (mon.0) 755 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:50:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:02.243938+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/320252880,v1:172.21.15.204:6801/320252880]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:50:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:02.244403+0000 mon.a (mon.0) 755 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T09:50:03.081 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:50:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:50:02.628+0000 7fe258a13700 -1 osd.4 93 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:01.969692+0000 mgr.y (mgr.25138) 251 : cluster [DBG] pgmap v142: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 101/609 objects degraded (16.585%) 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:02.583891+0000 mon.a (mon.0) 756 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:02.594683+0000 mon.a (mon.0) 757 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:02.594792+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e96: 8 total, 7 up, 8 in 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:02.596411+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/320252880,v1:172.21.15.204:6801/320252880]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:02.596783+0000 mon.a (mon.0) 759 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:01.969692+0000 mgr.y (mgr.25138) 251 : cluster [DBG] pgmap v142: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 1.1 GiB used, 710 GiB / 715 GiB avail; 101/609 objects degraded (16.585%) 2024-02-10T09:50:03.860 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:02.583891+0000 mon.a (mon.0) 756 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T09:50:03.861 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:02.594683+0000 mon.a (mon.0) 757 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T09:50:03.861 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:02.594792+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e96: 8 total, 7 up, 8 in 2024-02-10T09:50:03.861 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:02.596411+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.204:6800/320252880,v1:172.21.15.204:6801/320252880]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:03.861 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:02.596783+0000 mon.a (mon.0) 759 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:50:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:50:04.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:50:03.858+0000 7fc4ea3a1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-10T09:50:04.619 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:04] "GET /metrics HTTP/1.1" 200 164023 "" "Prometheus/2.33.4" 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:03.591196+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:03.594805+0000 mon.a (mon.0) 761 : cluster [INF] osd.4 [v2:172.21.15.204:6800/320252880,v1:172.21.15.204:6801/320252880] boot 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:03.594858+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.595888+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.842998+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.847995+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.853033+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.855416+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:04.620 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.856773+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.857838+0000 mon.a (mon.0) 769 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.859254+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:03.859406+0000 mgr.y (mgr.25138) 252 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:03.860234+0000 mgr.y (mgr.25138) 253 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:03.970165+0000 mgr.y (mgr.25138) 254 : cluster [DBG] pgmap v145: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 101/609 objects degraded (16.585%) 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:04.599217+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:03.591196+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:03.594805+0000 mon.a (mon.0) 761 : cluster [INF] osd.4 [v2:172.21.15.204:6800/320252880,v1:172.21.15.204:6801/320252880] boot 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:03.594858+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.595888+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T09:50:04.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.842998+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.847995+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.853033+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.855416+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.856773+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.857838+0000 mon.a (mon.0) 769 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.859254+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:03.859406+0000 mgr.y (mgr.25138) 252 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:03.860234+0000 mgr.y (mgr.25138) 253 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:03.970165+0000 mgr.y (mgr.25138) 254 : cluster [DBG] pgmap v145: 129 pgs: 22 active+undersized+degraded, 34 active+undersized, 73 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 101/609 objects degraded (16.585%) 2024-02-10T09:50:04.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:04.599217+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-10T09:50:06.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:05.109038+0000 mon.a (mon.0) 772 : cluster [WRN] Health check update: Degraded data redundancy: 101/609 objects degraded (16.585%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:06.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:05.109038+0000 mon.a (mon.0) 772 : cluster [WRN] Health check update: Degraded data redundancy: 101/609 objects degraded (16.585%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:07.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:05.970928+0000 mgr.y (mgr.25138) 255 : cluster [DBG] pgmap v147: 129 pgs: 7 peering, 20 active+undersized+degraded, 26 active+undersized, 76 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 90/609 objects degraded (14.778%) 2024-02-10T09:50:07.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:05.970928+0000 mgr.y (mgr.25138) 255 : cluster [DBG] pgmap v147: 129 pgs: 7 peering, 20 active+undersized+degraded, 26 active+undersized, 76 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 90/609 objects degraded (14.778%) 2024-02-10T09:50:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:07.971695+0000 mgr.y (mgr.25138) 256 : cluster [DBG] pgmap v148: 129 pgs: 7 peering, 9 active+undersized+degraded, 11 active+undersized, 102 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 49/609 objects degraded (8.046%) 2024-02-10T09:50:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:08.079830+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:09.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:08.219660+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:09.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:07.971695+0000 mgr.y (mgr.25138) 256 : cluster [DBG] pgmap v148: 129 pgs: 7 peering, 9 active+undersized+degraded, 11 active+undersized, 102 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 49/609 objects degraded (8.046%) 2024-02-10T09:50:09.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:08.079830+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:09.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:08.219660+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:10.079124+0000 mon.a (mon.0) 775 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/609 objects degraded (8.046%), 9 pgs degraded) 2024-02-10T09:50:10.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:10.079180+0000 mon.a (mon.0) 776 : cluster [INF] Cluster is now healthy 2024-02-10T09:50:10.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:10.079124+0000 mon.a (mon.0) 775 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/609 objects degraded (8.046%), 9 pgs degraded) 2024-02-10T09:50:10.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:10.079180+0000 mon.a (mon.0) 776 : cluster [INF] Cluster is now healthy 2024-02-10T09:50:11.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:09.972371+0000 mgr.y (mgr.25138) 257 : cluster [DBG] pgmap v149: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:11.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:09.972371+0000 mgr.y (mgr.25138) 257 : cluster [DBG] pgmap v149: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:13.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:11.972960+0000 mgr.y (mgr.25138) 258 : cluster [DBG] pgmap v150: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:13.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:11.972960+0000 mgr.y (mgr.25138) 258 : cluster [DBG] pgmap v150: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:50:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:50:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:14] "GET /metrics HTTP/1.1" 200 164780 "" "Prometheus/2.33.4" 2024-02-10T09:50:15.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:13.973623+0000 mgr.y (mgr.25138) 259 : cluster [DBG] pgmap v151: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:15.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:13.973623+0000 mgr.y (mgr.25138) 259 : cluster [DBG] pgmap v151: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:17.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:15.974367+0000 mgr.y (mgr.25138) 260 : cluster [DBG] pgmap v152: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:17.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:15.974367+0000 mgr.y (mgr.25138) 260 : cluster [DBG] pgmap v152: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:17.974927+0000 mgr.y (mgr.25138) 261 : cluster [DBG] pgmap v153: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:18.231042+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:18.874620+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:19.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:17.974927+0000 mgr.y (mgr.25138) 261 : cluster [DBG] pgmap v153: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:18.231042+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:18.874620+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:20.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:18.874831+0000 mgr.y (mgr.25138) 262 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:20.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:18.875973+0000 mgr.y (mgr.25138) 263 : cephadm [INF] Upgrade: osd.5 is safe to restart 2024-02-10T09:50:20.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:18.874831+0000 mgr.y (mgr.25138) 262 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T09:50:20.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:18.875973+0000 mgr.y (mgr.25138) 263 : cephadm [INF] Upgrade: osd.5 is safe to restart 2024-02-10T09:50:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:19.975497+0000 mgr.y (mgr.25138) 264 : cluster [DBG] pgmap v154: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:20.389261+0000 mgr.y (mgr.25138) 265 : cephadm [INF] Upgrade: Updating osd.5 2024-02-10T09:50:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:20.393192+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:20.393838+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:50:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:20.394318+0000 mon.a (mon.0) 781 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:50:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:20.395341+0000 mgr.y (mgr.25138) 266 : cephadm [INF] Deploying daemon osd.5 on smithi204 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:20.395670+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "container_image"}]: dispatch 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:19.975497+0000 mgr.y (mgr.25138) 264 : cluster [DBG] pgmap v154: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:20.389261+0000 mgr.y (mgr.25138) 265 : cephadm [INF] Upgrade: Updating osd.5 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:20.393192+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:20.393838+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:20.394318+0000 mon.a (mon.0) 781 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:20.395341+0000 mgr.y (mgr.25138) 266 : cephadm [INF] Deploying daemon osd.5 on smithi204 2024-02-10T09:50:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:20.395670+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "container_image"}]: dispatch 2024-02-10T09:50:23.582 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:23 smithi204 systemd[1]: Stopping Ceph osd.5 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:50:23.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:21.976168+0000 mgr.y (mgr.25138) 267 : cluster [DBG] pgmap v155: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:23.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:23.080006+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:23.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:21.976168+0000 mgr.y (mgr.25138) 267 : cluster [DBG] pgmap v155: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:23.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:23.080006+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:23.917 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:50:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:50:23.918 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:23 smithi204 bash[39725]: debug 2024-02-10T09:50:23.585+0000 7f663a73e700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:50:23.918 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:23 smithi204 bash[39725]: debug 2024-02-10T09:50:23.585+0000 7f663a73e700 -1 osd.5 98 *** Got signal Terminated *** 2024-02-10T09:50:23.918 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:23 smithi204 bash[39725]: debug 2024-02-10T09:50:23.585+0000 7f663a73e700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:50:24.195 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:24 smithi204 bash[64353]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.5 2024-02-10T09:50:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:24] "GET /metrics HTTP/1.1" 200 164822 "" "Prometheus/2.33.4" 2024-02-10T09:50:24.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.637911+0000 mon.a (mon.0) 784 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.637956+0000 mon.a (mon.0) 785 : cluster [INF] osd.5 failed (root=default,host=smithi204) (connection refused reported by osd.1) 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.638163+0000 mon.a (mon.0) 786 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.638304+0000 mon.a (mon.0) 787 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.638453+0000 mon.a (mon.0) 788 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.838145+0000 mon.a (mon.0) 789 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.838206+0000 mon.a (mon.0) 790 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.838491+0000 mon.a (mon.0) 791 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.838699+0000 mon.a (mon.0) 792 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.838863+0000 mon.a (mon.0) 793 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839019+0000 mon.a (mon.0) 794 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839152+0000 mon.a (mon.0) 795 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839315+0000 mon.a (mon.0) 796 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839449+0000 mon.a (mon.0) 797 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839604+0000 mon.a (mon.0) 798 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839750+0000 mon.a (mon.0) 799 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.839905+0000 mon.a (mon.0) 800 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.840023+0000 mon.a (mon.0) 801 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.840137+0000 mon.a (mon.0) 802 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.239169+0000 mon.a (mon.0) 803 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.637911+0000 mon.a (mon.0) 784 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.637956+0000 mon.a (mon.0) 785 : cluster [INF] osd.5 failed (root=default,host=smithi204) (connection refused reported by osd.1) 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.638163+0000 mon.a (mon.0) 786 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.638304+0000 mon.a (mon.0) 787 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.638453+0000 mon.a (mon.0) 788 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.838145+0000 mon.a (mon.0) 789 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.838206+0000 mon.a (mon.0) 790 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.838491+0000 mon.a (mon.0) 791 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.838699+0000 mon.a (mon.0) 792 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.838863+0000 mon.a (mon.0) 793 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839019+0000 mon.a (mon.0) 794 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839152+0000 mon.a (mon.0) 795 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839315+0000 mon.a (mon.0) 796 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839449+0000 mon.a (mon.0) 797 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839604+0000 mon.a (mon.0) 798 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839750+0000 mon.a (mon.0) 799 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.839905+0000 mon.a (mon.0) 800 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.840023+0000 mon.a (mon.0) 801 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.840137+0000 mon.a (mon.0) 802 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.239169+0000 mon.a (mon.0) 803 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.239242+0000 mon.a (mon.0) 804 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.239556+0000 mon.a (mon.0) 805 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.239714+0000 mon.a (mon.0) 806 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.239893+0000 mon.a (mon.0) 807 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240047+0000 mon.a (mon.0) 808 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240180+0000 mon.a (mon.0) 809 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240339+0000 mon.a (mon.0) 810 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240460+0000 mon.a (mon.0) 811 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240614+0000 mon.a (mon.0) 812 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240751+0000 mon.a (mon.0) 813 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.240909+0000 mon.a (mon.0) 814 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.241026+0000 mon.a (mon.0) 815 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.241139+0000 mon.a (mon.0) 816 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.239242+0000 mon.a (mon.0) 804 : cluster [DBG] osd.5 reported immediately failed by osd.3 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.239556+0000 mon.a (mon.0) 805 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.239714+0000 mon.a (mon.0) 806 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.239893+0000 mon.a (mon.0) 807 : cluster [DBG] osd.5 reported immediately failed by osd.2 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240047+0000 mon.a (mon.0) 808 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240180+0000 mon.a (mon.0) 809 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240339+0000 mon.a (mon.0) 810 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240460+0000 mon.a (mon.0) 811 : cluster [DBG] osd.5 reported immediately failed by osd.1 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240614+0000 mon.a (mon.0) 812 : cluster [DBG] osd.5 reported immediately failed by osd.6 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240751+0000 mon.a (mon.0) 813 : cluster [DBG] osd.5 reported immediately failed by osd.0 2024-02-10T09:50:24.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.240909+0000 mon.a (mon.0) 814 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:24.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.241026+0000 mon.a (mon.0) 815 : cluster [DBG] osd.5 reported immediately failed by osd.7 2024-02-10T09:50:24.695 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.241139+0000 mon.a (mon.0) 816 : cluster [DBG] osd.5 reported immediately failed by osd.4 2024-02-10T09:50:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:23.976979+0000 mgr.y (mgr.25138) 268 : cluster [DBG] pgmap v156: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.395962+0000 mon.a (mon.0) 817 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:50:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:24.410007+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e99: 8 total, 7 up, 8 in 2024-02-10T09:50:25.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:23.976979+0000 mgr.y (mgr.25138) 268 : cluster [DBG] pgmap v156: 129 pgs: 129 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:25.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.395962+0000 mon.a (mon.0) 817 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:50:25.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:24.410007+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e99: 8 total, 7 up, 8 in 2024-02-10T09:50:26.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:25 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.5.service: Succeeded. 2024-02-10T09:50:26.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:25 smithi204 systemd[1]: Stopped Ceph osd.5 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:50:26.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 systemd[1]: Starting Ceph osd.5 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:50:26.636 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:50:26.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:25.411151+0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e100: 8 total, 7 up, 8 in 2024-02-10T09:50:26.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:25.411151+0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e100: 8 total, 7 up, 8 in 2024-02-10T09:50:27.253 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:50:27.253 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (18m) 52s ago 19m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:50:27.253 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (6m) 23s ago 21m 57.7M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:50:27.253 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (6m) 23s ago 31m 386M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (14m) 52s ago 33m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (4m) 52s ago 33m 84.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (5m) 23s ago 32m 95.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (4m) 52s ago 32m 69.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (6m) 52s ago 24m 21.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (6m) 23s ago 24m 23.2M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 52s ago 31m 40.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 52s ago 31m 40.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (105s) 52s ago 30m 40.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (56s) 52s ago 30m 10.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (28s) 23s ago 30m 10.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (29m) 23s ago 29m 52.9M 4096M 15.2.0 204a01f9b0b6 5d6ec7ebaa7b 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (29m) 23s ago 29m 51.8M 4096M 15.2.0 204a01f9b0b6 cb0fc1977562 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (29m) 23s ago 29m 51.2M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:50:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (18m) 23s ago 18m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:50:27.255 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (21m) 52s ago 21m 39.1M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:50:27.255 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (21m) 23s ago 21m 39.2M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T09:50:27.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:26 smithi204 bash[64657]: --> ceph-volume lvm activate successful for osd ID: 5 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:50:27.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2, 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 5 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 4, 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 10 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:50:27.627 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:50:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:25.977598+0000 mgr.y (mgr.25138) 269 : cluster [DBG] pgmap v159: 129 pgs: 6 peering, 13 stale+active+clean, 110 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:26.408377+0000 mon.a (mon.0) 820 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-02-10T09:50:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.244007+0000 mon.a (mon.0) 821 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.244329+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.244594+0000 mon.a (mon.0) 823 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.244873+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.245128+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.245381+0000 mon.a (mon.0) 826 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.246970+0000 mon.a (mon.0) 827 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.247244+0000 mon.a (mon.0) 828 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.247503+0000 mon.a (mon.0) 829 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.247765+0000 mon.a (mon.0) 830 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.248019+0000 mon.a (mon.0) 831 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:25.977598+0000 mgr.y (mgr.25138) 269 : cluster [DBG] pgmap v159: 129 pgs: 6 peering, 13 stale+active+clean, 110 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:26.408377+0000 mon.a (mon.0) 820 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-02-10T09:50:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.244007+0000 mon.a (mon.0) 821 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.244329+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.244594+0000 mon.a (mon.0) 823 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.244873+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.245128+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.245381+0000 mon.a (mon.0) 826 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.246970+0000 mon.a (mon.0) 827 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.247244+0000 mon.a (mon.0) 828 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.247503+0000 mon.a (mon.0) 829 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.247765+0000 mon.a (mon.0) 830 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.248019+0000 mon.a (mon.0) 831 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "5/8 daemons upgraded", 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:50:27.939 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:50:27.940 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:50:28.581 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:28 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:50:28.231+0000 7f52fbcc4700 1 -- 172.21.15.204:0/1321348442 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e35c7d0680 con 0x55e35c797000 2024-02-10T09:50:28.582 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:28 smithi204 bash[64925]: d9e9aab0c7537b6e473e20d8745ee3dd575e16c8582b5ef9618a11abdd3d8687 2024-02-10T09:50:28.582 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:28 smithi204 systemd[1]: Started Ceph osd.5 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:50:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:26.624084+0000 mgr.y (mgr.25138) 270 : audit [DBG] from='client.64381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:26.937825+0000 mgr.y (mgr.25138) 271 : audit [DBG] from='client.54438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.242146+0000 mgr.y (mgr.25138) 272 : audit [DBG] from='client.64393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.625526+0000 mon.a (mon.0) 832 : audit [DBG] from='client.64399 172.21.15.153:0/2896433685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:28.362600+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:28.363354+0000 mon.a (mon.0) 834 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:28.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:26.624084+0000 mgr.y (mgr.25138) 270 : audit [DBG] from='client.64381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:28.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:26.937825+0000 mgr.y (mgr.25138) 271 : audit [DBG] from='client.54438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:28.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.242146+0000 mgr.y (mgr.25138) 272 : audit [DBG] from='client.64393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:28.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.625526+0000 mon.a (mon.0) 832 : audit [DBG] from='client.64399 172.21.15.153:0/2896433685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:28.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:28.362600+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:28.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:28.363354+0000 mon.a (mon.0) 834 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:29.081 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:28 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:50:28.756+0000 7f5303752380 -1 Falling back to public interface 2024-02-10T09:50:29.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:27.938425+0000 mgr.y (mgr.25138) 273 : audit [DBG] from='client.64405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:29.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:27.978133+0000 mgr.y (mgr.25138) 274 : cluster [DBG] pgmap v160: 129 pgs: 5 active+undersized+degraded, 6 peering, 7 stale+active+clean, 11 active+undersized, 100 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 22/609 objects degraded (3.612%) 2024-02-10T09:50:29.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:28.414185+0000 mon.a (mon.0) 835 : cluster [WRN] Health check failed: Degraded data redundancy: 22/609 objects degraded (3.612%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:29.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:28.604053+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:29.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:28.604942+0000 mon.a (mon.0) 837 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "container_image"}]: dispatch 2024-02-10T09:50:29.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:28.607202+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:29.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:28.608149+0000 mon.a (mon.0) 839 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:27.938425+0000 mgr.y (mgr.25138) 273 : audit [DBG] from='client.64405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:27.978133+0000 mgr.y (mgr.25138) 274 : cluster [DBG] pgmap v160: 129 pgs: 5 active+undersized+degraded, 6 peering, 7 stale+active+clean, 11 active+undersized, 100 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 22/609 objects degraded (3.612%) 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:28.414185+0000 mon.a (mon.0) 835 : cluster [WRN] Health check failed: Degraded data redundancy: 22/609 objects degraded (3.612%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:28.604053+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:28.604942+0000 mon.a (mon.0) 837 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "container_image"}]: dispatch 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:28.607202+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:28.608149+0000 mon.a (mon.0) 839 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:31.659 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:29.978704+0000 mgr.y (mgr.25138) 275 : cluster [DBG] pgmap v161: 129 pgs: 16 active+undersized+degraded, 6 peering, 22 active+undersized, 85 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 54/609 objects degraded (8.867%) 2024-02-10T09:50:31.660 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:29.978704+0000 mgr.y (mgr.25138) 275 : cluster [DBG] pgmap v161: 129 pgs: 16 active+undersized+degraded, 6 peering, 22 active+undersized, 85 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 54/609 objects degraded (8.867%) 2024-02-10T09:50:31.831 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:31 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:50:31.480+0000 7f5303752380 -1 osd.5 98 log_to_monitors {default=true} 2024-02-10T09:50:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:31.484222+0000 mon.b (mon.2) 9 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/2061751638,v1:172.21.15.204:6809/2061751638]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:50:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:31.484658+0000 mon.a (mon.0) 840 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:50:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:31.678440+0000 mon.a (mon.0) 841 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:32.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:31.896844+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:31.484222+0000 mon.b (mon.2) 9 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/2061751638,v1:172.21.15.204:6809/2061751638]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:50:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:31.484658+0000 mon.a (mon.0) 840 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T09:50:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:31.678440+0000 mon.a (mon.0) 841 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:31.896844+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:32.831 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:50:32 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:50:32.451+0000 7f52f2c35700 -1 osd.5 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:50:33.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:50:33.403+0000 7fc4ea3a1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-10T09:50:33.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:31.979401+0000 mgr.y (mgr.25138) 276 : cluster [DBG] pgmap v162: 129 pgs: 21 active+undersized+degraded, 23 active+undersized, 85 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 73/609 objects degraded (11.987%) 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:32.433438+0000 mon.a (mon.0) 843 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:32.433510+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e101: 8 total, 7 up, 8 in 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:32.434467+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/2061751638,v1:172.21.15.204:6809/2061751638]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:32.434965+0000 mon.a (mon.0) 845 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:32.895036+0000 mon.a (mon.0) 846 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.114600+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.394311+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.398853+0000 mon.a (mon.0) 849 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.401126+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.402279+0000 mon.a (mon.0) 851 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.403246+0000 mon.a (mon.0) 852 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.404516+0000 mon.a (mon.0) 853 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:31.979401+0000 mgr.y (mgr.25138) 276 : cluster [DBG] pgmap v162: 129 pgs: 21 active+undersized+degraded, 23 active+undersized, 85 active+clean; 7.0 KiB data, 1.4 GiB used, 711 GiB / 715 GiB avail; 73/609 objects degraded (11.987%) 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:32.433438+0000 mon.a (mon.0) 843 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:32.433510+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e101: 8 total, 7 up, 8 in 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:32.434467+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.204:6808/2061751638,v1:172.21.15.204:6809/2061751638]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:32.434965+0000 mon.a (mon.0) 845 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:32.895036+0000 mon.a (mon.0) 846 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.114600+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.394311+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.398853+0000 mon.a (mon.0) 849 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.401126+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.402279+0000 mon.a (mon.0) 851 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.403246+0000 mon.a (mon.0) 852 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:33.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.404516+0000 mon.a (mon.0) 853 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:50:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:50:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:34] "GET /metrics HTTP/1.1" 200 164822 "" "Prometheus/2.33.4" 2024-02-10T09:50:34.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.404676+0000 mgr.y (mgr.25138) 277 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:34.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:33.405408+0000 mgr.y (mgr.25138) 278 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:33.431186+0000 mon.a (mon.0) 854 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:33.440524+0000 mon.a (mon.0) 855 : cluster [INF] osd.5 [v2:172.21.15.204:6808/2061751638,v1:172.21.15.204:6809/2061751638] boot 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:33.440571+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:33.444425+0000 mon.a (mon.0) 857 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:34.439589+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.404676+0000 mgr.y (mgr.25138) 277 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:33.405408+0000 mgr.y (mgr.25138) 278 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:33.431186+0000 mon.a (mon.0) 854 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:33.440524+0000 mon.a (mon.0) 855 : cluster [INF] osd.5 [v2:172.21.15.204:6808/2061751638,v1:172.21.15.204:6809/2061751638] boot 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:33.440571+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:33.444425+0000 mon.a (mon.0) 857 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T09:50:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:34.439589+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-10T09:50:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:33.979959+0000 mgr.y (mgr.25138) 279 : cluster [DBG] pgmap v165: 129 pgs: 21 active+undersized+degraded, 23 active+undersized, 85 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 73/609 objects degraded (11.987%) 2024-02-10T09:50:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:35.145527+0000 mon.a (mon.0) 859 : cluster [WRN] Health check update: Degraded data redundancy: 73/609 objects degraded (11.987%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:33.979959+0000 mgr.y (mgr.25138) 279 : cluster [DBG] pgmap v165: 129 pgs: 21 active+undersized+degraded, 23 active+undersized, 85 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 73/609 objects degraded (11.987%) 2024-02-10T09:50:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:35.145527+0000 mon.a (mon.0) 859 : cluster [WRN] Health check update: Degraded data redundancy: 73/609 objects degraded (11.987%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:37.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:35.980641+0000 mgr.y (mgr.25138) 280 : cluster [DBG] pgmap v167: 129 pgs: 6 peering, 17 active+undersized+degraded, 19 active+undersized, 87 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 57/609 objects degraded (9.360%) 2024-02-10T09:50:37.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:35.980641+0000 mgr.y (mgr.25138) 280 : cluster [DBG] pgmap v167: 129 pgs: 6 peering, 17 active+undersized+degraded, 19 active+undersized, 87 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 57/609 objects degraded (9.360%) 2024-02-10T09:50:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:37.981422+0000 mgr.y (mgr.25138) 281 : cluster [DBG] pgmap v168: 129 pgs: 6 peering, 6 active+undersized+degraded, 3 active+undersized, 114 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 16/609 objects degraded (2.627%) 2024-02-10T09:50:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:38.080317+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:38.248711+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:37.981422+0000 mgr.y (mgr.25138) 281 : cluster [DBG] pgmap v168: 129 pgs: 6 peering, 6 active+undersized+degraded, 3 active+undersized, 114 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 16/609 objects degraded (2.627%) 2024-02-10T09:50:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:38.080317+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:38.248711+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:40.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:40.079595+0000 mon.a (mon.0) 862 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/609 objects degraded (2.627%), 6 pgs degraded) 2024-02-10T09:50:40.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:40.079665+0000 mon.a (mon.0) 863 : cluster [INF] Cluster is now healthy 2024-02-10T09:50:40.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:40.079595+0000 mon.a (mon.0) 862 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/609 objects degraded (2.627%), 6 pgs degraded) 2024-02-10T09:50:40.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:40.079665+0000 mon.a (mon.0) 863 : cluster [INF] Cluster is now healthy 2024-02-10T09:50:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:39.982205+0000 mgr.y (mgr.25138) 282 : cluster [DBG] pgmap v169: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:41.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:39.982205+0000 mgr.y (mgr.25138) 282 : cluster [DBG] pgmap v169: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:43.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:41.982821+0000 mgr.y (mgr.25138) 283 : cluster [DBG] pgmap v170: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:43.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:41.982821+0000 mgr.y (mgr.25138) 283 : cluster [DBG] pgmap v170: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:50:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:50:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:44] "GET /metrics HTTP/1.1" 200 165573 "" "Prometheus/2.33.4" 2024-02-10T09:50:45.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:43.983470+0000 mgr.y (mgr.25138) 284 : cluster [DBG] pgmap v171: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:45.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:43.983470+0000 mgr.y (mgr.25138) 284 : cluster [DBG] pgmap v171: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:47.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:45.984052+0000 mgr.y (mgr.25138) 285 : cluster [DBG] pgmap v172: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:47.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:45.984052+0000 mgr.y (mgr.25138) 285 : cluster [DBG] pgmap v172: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:49.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:47.984739+0000 mgr.y (mgr.25138) 286 : cluster [DBG] pgmap v173: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:49.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:48.260561+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:49.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:48.420989+0000 mon.a (mon.0) 865 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:48.421197+0000 mgr.y (mgr.25138) 287 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:48.422290+0000 mgr.y (mgr.25138) 288 : cephadm [INF] Upgrade: osd.6 is safe to restart 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:47.984739+0000 mgr.y (mgr.25138) 286 : cluster [DBG] pgmap v173: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:48.260561+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:48.420989+0000 mon.a (mon.0) 865 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:48.421197+0000 mgr.y (mgr.25138) 287 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T09:50:49.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:48.422290+0000 mgr.y (mgr.25138) 288 : cephadm [INF] Upgrade: osd.6 is safe to restart 2024-02-10T09:50:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:49.821675+0000 mgr.y (mgr.25138) 289 : cephadm [INF] Upgrade: Updating osd.6 2024-02-10T09:50:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:49.829082+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:49.830014+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:50:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:49.830723+0000 mon.a (mon.0) 868 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:50:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:50:49.832104+0000 mgr.y (mgr.25138) 290 : cephadm [INF] Deploying daemon osd.6 on smithi204 2024-02-10T09:50:51.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:49.832646+0000 mon.a (mon.0) 869 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "container_image"}]: dispatch 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:49.985455+0000 mgr.y (mgr.25138) 291 : cluster [DBG] pgmap v174: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:49.821675+0000 mgr.y (mgr.25138) 289 : cephadm [INF] Upgrade: Updating osd.6 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:49.829082+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:49.830014+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:49.830723+0000 mon.a (mon.0) 868 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:50:49.832104+0000 mgr.y (mgr.25138) 290 : cephadm [INF] Deploying daemon osd.6 on smithi204 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:49.832646+0000 mon.a (mon.0) 869 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "container_image"}]: dispatch 2024-02-10T09:50:51.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:49.985455+0000 mgr.y (mgr.25138) 291 : cluster [DBG] pgmap v174: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:52.727 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:52 smithi204 systemd[1]: Stopping Ceph osd.6 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:50:53.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:52 smithi204 bash[41390]: debug 2024-02-10T09:50:52.724+0000 7ff4747c2700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:50:53.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:52 smithi204 bash[41390]: debug 2024-02-10T09:50:52.724+0000 7ff4747c2700 -1 osd.6 103 *** Got signal Terminated *** 2024-02-10T09:50:53.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:52 smithi204 bash[41390]: debug 2024-02-10T09:50:52.724+0000 7ff4747c2700 -1 osd.6 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:50:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:51.985969+0000 mgr.y (mgr.25138) 292 : cluster [DBG] pgmap v175: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.970180+0000 mon.a (mon.0) 870 : cluster [DBG] osd.6 reported immediately failed by osd.3 2024-02-10T09:50:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.970223+0000 mon.a (mon.0) 871 : cluster [INF] osd.6 failed (root=default,host=smithi204) (connection refused reported by osd.3) 2024-02-10T09:50:53.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.970447+0000 mon.a (mon.0) 872 : cluster [DBG] osd.6 reported immediately failed by osd.3 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.970646+0000 mon.a (mon.0) 873 : cluster [DBG] osd.6 reported immediately failed by osd.2 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.970874+0000 mon.a (mon.0) 874 : cluster [DBG] osd.6 reported immediately failed by osd.2 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.971060+0000 mon.a (mon.0) 875 : cluster [DBG] osd.6 reported immediately failed by osd.7 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.971241+0000 mon.a (mon.0) 876 : cluster [DBG] osd.6 reported immediately failed by osd.0 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.971424+0000 mon.a (mon.0) 877 : cluster [DBG] osd.6 reported immediately failed by osd.4 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.971607+0000 mon.a (mon.0) 878 : cluster [DBG] osd.6 reported immediately failed by osd.1 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.971802+0000 mon.a (mon.0) 879 : cluster [DBG] osd.6 reported immediately failed by osd.5 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.971991+0000 mon.a (mon.0) 880 : cluster [DBG] osd.6 reported immediately failed by osd.0 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.972175+0000 mon.a (mon.0) 881 : cluster [DBG] osd.6 reported immediately failed by osd.7 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.972361+0000 mon.a (mon.0) 882 : cluster [DBG] osd.6 reported immediately failed by osd.1 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.972546+0000 mon.a (mon.0) 883 : cluster [DBG] osd.6 reported immediately failed by osd.4 2024-02-10T09:50:53.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:52.972752+0000 mon.a (mon.0) 884 : cluster [DBG] osd.6 reported immediately failed by osd.5 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:51.985969+0000 mgr.y (mgr.25138) 292 : cluster [DBG] pgmap v175: 129 pgs: 129 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.970180+0000 mon.a (mon.0) 870 : cluster [DBG] osd.6 reported immediately failed by osd.3 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.970223+0000 mon.a (mon.0) 871 : cluster [INF] osd.6 failed (root=default,host=smithi204) (connection refused reported by osd.3) 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.970447+0000 mon.a (mon.0) 872 : cluster [DBG] osd.6 reported immediately failed by osd.3 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.970646+0000 mon.a (mon.0) 873 : cluster [DBG] osd.6 reported immediately failed by osd.2 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.970874+0000 mon.a (mon.0) 874 : cluster [DBG] osd.6 reported immediately failed by osd.2 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.971060+0000 mon.a (mon.0) 875 : cluster [DBG] osd.6 reported immediately failed by osd.7 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.971241+0000 mon.a (mon.0) 876 : cluster [DBG] osd.6 reported immediately failed by osd.0 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.971424+0000 mon.a (mon.0) 877 : cluster [DBG] osd.6 reported immediately failed by osd.4 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.971607+0000 mon.a (mon.0) 878 : cluster [DBG] osd.6 reported immediately failed by osd.1 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.971802+0000 mon.a (mon.0) 879 : cluster [DBG] osd.6 reported immediately failed by osd.5 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.971991+0000 mon.a (mon.0) 880 : cluster [DBG] osd.6 reported immediately failed by osd.0 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.972175+0000 mon.a (mon.0) 881 : cluster [DBG] osd.6 reported immediately failed by osd.7 2024-02-10T09:50:53.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.972361+0000 mon.a (mon.0) 882 : cluster [DBG] osd.6 reported immediately failed by osd.1 2024-02-10T09:50:53.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.972546+0000 mon.a (mon.0) 883 : cluster [DBG] osd.6 reported immediately failed by osd.4 2024-02-10T09:50:53.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:52.972752+0000 mon.a (mon.0) 884 : cluster [DBG] osd.6 reported immediately failed by osd.5 2024-02-10T09:50:53.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:53 smithi204 bash[66080]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.6 2024-02-10T09:50:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:50:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:50:54.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:53 smithi204 bash[66135]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T09:50:54.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:53.038924+0000 mon.a (mon.0) 885 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:50:54.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:53.047273+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e104: 8 total, 7 up, 8 in 2024-02-10T09:50:54.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:53.080019+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:54.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:53.038924+0000 mon.a (mon.0) 885 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:50:54.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:53.047273+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e104: 8 total, 7 up, 8 in 2024-02-10T09:50:54.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:53.080019+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:50:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:50:54] "GET /metrics HTTP/1.1" 200 165612 "" "Prometheus/2.33.4" 2024-02-10T09:50:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:53.986454+0000 mgr.y (mgr.25138) 293 : cluster [DBG] pgmap v177: 129 pgs: 13 stale+active+clean, 116 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:55.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:54.048788+0000 mon.a (mon.0) 888 : cluster [DBG] osdmap e105: 8 total, 7 up, 8 in 2024-02-10T09:50:55.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:53.986454+0000 mgr.y (mgr.25138) 293 : cluster [DBG] pgmap v177: 129 pgs: 13 stale+active+clean, 116 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:50:55.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:54.048788+0000 mon.a (mon.0) 888 : cluster [DBG] osdmap e105: 8 total, 7 up, 8 in 2024-02-10T09:50:55.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:55 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.6.service: Succeeded. 2024-02-10T09:50:55.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:55 smithi204 systemd[1]: Stopped Ceph osd.6 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:50:55.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:55 smithi204 systemd[1]: Starting Ceph osd.6 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-6 --no-mon-config 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T09:50:56.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:56 smithi204 bash[66391]: --> ceph-volume lvm activate successful for osd ID: 6 2024-02-10T09:50:56.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:56.151636+0000 mon.a (mon.0) 889 : cluster [WRN] Health check failed: Degraded data redundancy: 19/609 objects degraded (3.120%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:56.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:56 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:56.151636+0000 mon.a (mon.0) 889 : cluster [WRN] Health check failed: Degraded data redundancy: 19/609 objects degraded (3.120%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T09:50:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:55.987286+0000 mgr.y (mgr.25138) 294 : cluster [DBG] pgmap v179: 129 pgs: 3 active+undersized+degraded, 11 stale+active+clean, 8 active+undersized, 107 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 19/609 objects degraded (3.120%) 2024-02-10T09:50:57.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:55.987286+0000 mgr.y (mgr.25138) 294 : cluster [DBG] pgmap v179: 129 pgs: 3 active+undersized+degraded, 11 stale+active+clean, 8 active+undersized, 107 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 19/609 objects degraded (3.120%) 2024-02-10T09:50:57.831 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:50:57.447+0000 7f04acd55700 1 -- 172.21.15.204:0/1308804422 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556900c2a680 con 0x556900bf1400 2024-02-10T09:50:57.831 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:57 smithi204 bash[66633]: 7e67b938cc8805eae4e7712089f60fe181c0c74031513d0947e862d51cc7b13e 2024-02-10T09:50:57.832 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:57 smithi204 systemd[1]: Started Ceph osd.6 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:50:58.265 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:50:58.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:50:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:50:57.966+0000 7f04b47e3380 -1 Falling back to public interface 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (18m) 84s ago 19m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (6m) 25s ago 21m 57.8M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (7m) 25s ago 32m 386M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (15m) 84s ago 34m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (5m) 84s ago 34m 84.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (5m) 25s ago 32m 97.2M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (4m) 84s ago 32m 69.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (7m) 84s ago 24m 21.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (7m) 25s ago 24m 22.5M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 84s ago 32m 40.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (3m) 84s ago 31m 40.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:50:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 84s ago 31m 40.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (88s) 84s ago 31m 10.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (60s) 25s ago 30m 39.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (31s) 25s ago 30m 14.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 starting - - - 4096M 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (29m) 25s ago 29m 52.3M 4096M 15.2.0 204a01f9b0b6 553a3da04795 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (19m) 25s ago 19m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (21m) 84s ago 21m 39.1M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:50:58.887 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (21m) 25s ago 21m 39.4M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:50:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:57.579197+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:57.580253+0000 mon.a (mon.0) 891 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:58.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:57.582501+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:57.579197+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:50:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:57.580253+0000 mon.a (mon.0) 891 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:50:58.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:57.582501+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:50:59.259 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 1, 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 6 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:50:59.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:50:59.261 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:50:59.261 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3, 2024-02-10T09:50:59.261 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 11 2024-02-10T09:50:59.261 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:50:59.261 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:50:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:50:59.571 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:50:59.571 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:50:59.571 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T09:50:59.572 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:50:59.572 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "6/8 daemons upgraded", 2024-02-10T09:50:59.572 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:50:59.572 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:50:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:50:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:57.987898+0000 mgr.y (mgr.25138) 295 : cluster [DBG] pgmap v180: 129 pgs: 13 active+undersized+degraded, 1 stale+active+clean, 23 active+undersized, 92 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 58/609 objects degraded (9.524%) 2024-02-10T09:50:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.252306+0000 mgr.y (mgr.25138) 296 : audit [DBG] from='client.64417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.567484+0000 mgr.y (mgr.25138) 297 : audit [DBG] from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.876726+0000 mon.a (mon.0) 893 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.877053+0000 mon.a (mon.0) 894 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.877297+0000 mon.a (mon.0) 895 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.877548+0000 mon.a (mon.0) 896 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.877791+0000 mon.a (mon.0) 897 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.878028+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.879440+0000 mon.a (mon.0) 899 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.879681+0000 mon.a (mon.0) 900 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.879929+0000 mon.a (mon.0) 901 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.880162+0000 mon.a (mon.0) 902 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.880395+0000 mon.a (mon.0) 903 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:59.259276+0000 mon.a (mon.0) 904 : audit [DBG] from='client.54486 172.21.15.153:0/4221367403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:57.987898+0000 mgr.y (mgr.25138) 295 : cluster [DBG] pgmap v180: 129 pgs: 13 active+undersized+degraded, 1 stale+active+clean, 23 active+undersized, 92 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 58/609 objects degraded (9.524%) 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.252306+0000 mgr.y (mgr.25138) 296 : audit [DBG] from='client.64417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.567484+0000 mgr.y (mgr.25138) 297 : audit [DBG] from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.876726+0000 mon.a (mon.0) 893 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.877053+0000 mon.a (mon.0) 894 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.877297+0000 mon.a (mon.0) 895 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.877548+0000 mon.a (mon.0) 896 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.877791+0000 mon.a (mon.0) 897 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.878028+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.879440+0000 mon.a (mon.0) 899 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.879681+0000 mon.a (mon.0) 900 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.879929+0000 mon.a (mon.0) 901 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.880162+0000 mon.a (mon.0) 902 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.880395+0000 mon.a (mon.0) 903 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:50:59.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:50:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:59.259276+0000 mon.a (mon.0) 904 : audit [DBG] from='client.54486 172.21.15.153:0/4221367403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:58.874823+0000 mgr.y (mgr.25138) 298 : audit [DBG] from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:00.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:50:59.570820+0000 mgr.y (mgr.25138) 299 : audit [DBG] from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:00.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:58.874823+0000 mgr.y (mgr.25138) 298 : audit [DBG] from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:00.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:00 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:50:59.570820+0000 mgr.y (mgr.25138) 299 : audit [DBG] from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:01.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:51:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:51:00.713+0000 7f04b47e3380 -1 osd.6 103 log_to_monitors {default=true} 2024-02-10T09:51:01.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:50:59.988544+0000 mgr.y (mgr.25138) 300 : cluster [DBG] pgmap v181: 129 pgs: 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 72/609 objects degraded (11.823%) 2024-02-10T09:51:01.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:00.716176+0000 mon.b (mon.2) 11 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/376123891,v1:172.21.15.204:6817/376123891]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:51:01.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:00.716570+0000 mon.a (mon.0) 905 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:51:01.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:50:59.988544+0000 mgr.y (mgr.25138) 300 : cluster [DBG] pgmap v181: 129 pgs: 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 7.0 KiB data, 1.7 GiB used, 712 GiB / 715 GiB avail; 72/609 objects degraded (11.823%) 2024-02-10T09:51:01.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:00.716176+0000 mon.b (mon.2) 11 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/376123891,v1:172.21.15.204:6817/376123891]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:51:01.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:00.716570+0000 mon.a (mon.0) 905 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T09:51:02.081 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:51:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:51:01.616+0000 7f04a34c5700 -1 osd.6 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:51:02.615 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:51:02.205+0000 7fc4ea3a1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-10T09:51:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:01.596285+0000 mon.a (mon.0) 906 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T09:51:02.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:01.596368+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e106: 8 total, 7 up, 8 in 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:01.596932+0000 mon.b (mon.2) 12 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/376123891,v1:172.21.15.204:6817/376123891]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:01.597335+0000 mon.a (mon.0) 908 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.189935+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.194841+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.199952+0000 mon.a (mon.0) 911 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.202329+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.203710+0000 mon.a (mon.0) 913 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.204721+0000 mon.a (mon.0) 914 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.206208+0000 mon.a (mon.0) 915 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:01.596285+0000 mon.a (mon.0) 906 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T09:51:02.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:01.596368+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e106: 8 total, 7 up, 8 in 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:01.596932+0000 mon.b (mon.2) 12 : audit [INF] from='osd.6 [v2:172.21.15.204:6816/376123891,v1:172.21.15.204:6817/376123891]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:01.597335+0000 mon.a (mon.0) 908 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.189935+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.194841+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.199952+0000 mon.a (mon.0) 911 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.202329+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.203710+0000 mon.a (mon.0) 913 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.204721+0000 mon.a (mon.0) 914 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:02.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.206208+0000 mon.a (mon.0) 915 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:01.989113+0000 mgr.y (mgr.25138) 301 : cluster [DBG] pgmap v183: 129 pgs: 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 72/609 objects degraded (11.823%) 2024-02-10T09:51:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.206367+0000 mgr.y (mgr.25138) 302 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:02.207134+0000 mgr.y (mgr.25138) 303 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:02.594020+0000 mon.a (mon.0) 916 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:02.603971+0000 mon.a (mon.0) 917 : cluster [INF] osd.6 [v2:172.21.15.204:6816/376123891,v1:172.21.15.204:6817/376123891] boot 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:02.604022+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:02.608650+0000 mon.a (mon.0) 919 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:03.602249+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:01.989113+0000 mgr.y (mgr.25138) 301 : cluster [DBG] pgmap v183: 129 pgs: 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 72/609 objects degraded (11.823%) 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.206367+0000 mgr.y (mgr.25138) 302 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:02.207134+0000 mgr.y (mgr.25138) 303 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-10T09:51:03.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:02.594020+0000 mon.a (mon.0) 916 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:51:03.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:02.603971+0000 mon.a (mon.0) 917 : cluster [INF] osd.6 [v2:172.21.15.204:6816/376123891,v1:172.21.15.204:6817/376123891] boot 2024-02-10T09:51:03.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:02.604022+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-10T09:51:03.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:02.608650+0000 mon.a (mon.0) 919 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T09:51:03.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:03.602249+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-10T09:51:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:51:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:51:04.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:03.989743+0000 mgr.y (mgr.25138) 304 : cluster [DBG] pgmap v186: 129 pgs: 14 peering, 10 active+undersized+degraded, 19 active+undersized, 86 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 44/609 objects degraded (7.225%) 2024-02-10T09:51:04.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:03.989743+0000 mgr.y (mgr.25138) 304 : cluster [DBG] pgmap v186: 129 pgs: 14 peering, 10 active+undersized+degraded, 19 active+undersized, 86 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 44/609 objects degraded (7.225%) 2024-02-10T09:51:04.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:04] "GET /metrics HTTP/1.1" 200 165612 "" "Prometheus/2.33.4" 2024-02-10T09:51:05.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:04.614851+0000 mon.a (mon.0) 921 : cluster [WRN] Health check update: Degraded data redundancy: 44/609 objects degraded (7.225%), 10 pgs degraded (PG_DEGRADED) 2024-02-10T09:51:05.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:04.614851+0000 mon.a (mon.0) 921 : cluster [WRN] Health check update: Degraded data redundancy: 44/609 objects degraded (7.225%), 10 pgs degraded (PG_DEGRADED) 2024-02-10T09:51:06.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:05.990382+0000 mgr.y (mgr.25138) 305 : cluster [DBG] pgmap v187: 129 pgs: 14 peering, 7 active+undersized+degraded, 13 active+undersized, 95 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 25/609 objects degraded (4.105%) 2024-02-10T09:51:06.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:06 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:05.990382+0000 mgr.y (mgr.25138) 305 : cluster [DBG] pgmap v187: 129 pgs: 14 peering, 7 active+undersized+degraded, 13 active+undersized, 95 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 25/609 objects degraded (4.105%) 2024-02-10T09:51:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:08.041913+0000 mon.a (mon.0) 922 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/609 objects degraded (4.105%), 7 pgs degraded) 2024-02-10T09:51:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:08.041967+0000 mon.a (mon.0) 923 : cluster [INF] Cluster is now healthy 2024-02-10T09:51:08.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:08.082639+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:08.041913+0000 mon.a (mon.0) 922 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/609 objects degraded (4.105%), 7 pgs degraded) 2024-02-10T09:51:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:08.041967+0000 mon.a (mon.0) 923 : cluster [INF] Cluster is now healthy 2024-02-10T09:51:08.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:08.082639+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:07.991069+0000 mgr.y (mgr.25138) 306 : cluster [DBG] pgmap v188: 129 pgs: 6 peering, 123 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:08.275391+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:07.991069+0000 mgr.y (mgr.25138) 306 : cluster [DBG] pgmap v188: 129 pgs: 6 peering, 123 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:08.275391+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:11.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:09.991573+0000 mgr.y (mgr.25138) 307 : cluster [DBG] pgmap v189: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:11.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:09.991573+0000 mgr.y (mgr.25138) 307 : cluster [DBG] pgmap v189: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:13.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:11.992356+0000 mgr.y (mgr.25138) 308 : cluster [DBG] pgmap v190: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:11.992356+0000 mgr.y (mgr.25138) 308 : cluster [DBG] pgmap v190: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:51:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:51:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:14] "GET /metrics HTTP/1.1" 200 166362 "" "Prometheus/2.33.4" 2024-02-10T09:51:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:13.992944+0000 mgr.y (mgr.25138) 309 : cluster [DBG] pgmap v191: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:15.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:13.992944+0000 mgr.y (mgr.25138) 309 : cluster [DBG] pgmap v191: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:17.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:15.993677+0000 mgr.y (mgr.25138) 310 : cluster [DBG] pgmap v192: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:17.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:17.215663+0000 mon.a (mon.0) 926 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:15.993677+0000 mgr.y (mgr.25138) 310 : cluster [DBG] pgmap v192: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:17.215663+0000 mon.a (mon.0) 926 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:17.215860+0000 mgr.y (mgr.25138) 311 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:17.217013+0000 mgr.y (mgr.25138) 312 : cephadm [INF] Upgrade: osd.7 is safe to restart 2024-02-10T09:51:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:18.288126+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:17.215860+0000 mgr.y (mgr.25138) 311 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T09:51:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:17.217013+0000 mgr.y (mgr.25138) 312 : cephadm [INF] Upgrade: osd.7 is safe to restart 2024-02-10T09:51:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:18.288126+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:17.994256+0000 mgr.y (mgr.25138) 313 : cluster [DBG] pgmap v193: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:18.580399+0000 mgr.y (mgr.25138) 314 : cephadm [INF] Upgrade: Updating osd.7 2024-02-10T09:51:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:18.586543+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:18.587430+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:51:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:18.588062+0000 mon.a (mon.0) 930 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:51:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:18.589329+0000 mgr.y (mgr.25138) 315 : cephadm [INF] Deploying daemon osd.7 on smithi204 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:18.589783+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "container_image"}]: dispatch 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:17.994256+0000 mgr.y (mgr.25138) 313 : cluster [DBG] pgmap v193: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:18.580399+0000 mgr.y (mgr.25138) 314 : cephadm [INF] Upgrade: Updating osd.7 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:18.586543+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:18.587430+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:18.588062+0000 mon.a (mon.0) 930 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:18.589329+0000 mgr.y (mgr.25138) 315 : cephadm [INF] Deploying daemon osd.7 on smithi204 2024-02-10T09:51:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:18.589783+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "container_image"}]: dispatch 2024-02-10T09:51:21.517 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:21 smithi204 systemd[1]: Stopping Ceph osd.7 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:51:21.806 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:21 smithi204 bash[42840]: debug 2024-02-10T09:51:21.515+0000 7fc0fb837700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2024-02-10T09:51:21.807 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:21 smithi204 bash[42840]: debug 2024-02-10T09:51:21.515+0000 7fc0fb837700 -1 osd.7 108 *** Got signal Terminated *** 2024-02-10T09:51:21.807 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:21 smithi204 bash[42840]: debug 2024-02-10T09:51:21.515+0000 7fc0fb837700 -1 osd.7 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:51:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:19.994977+0000 mgr.y (mgr.25138) 316 : cluster [DBG] pgmap v194: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:21.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:19.994977+0000 mgr.y (mgr.25138) 316 : cluster [DBG] pgmap v194: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:22.075 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:21 smithi204 bash[67722]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.7 2024-02-10T09:51:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.771185+0000 mon.a (mon.0) 932 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.771220+0000 mon.a (mon.0) 933 : cluster [INF] osd.7 failed (root=default,host=smithi204) (connection refused reported by osd.3) 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.771493+0000 mon.a (mon.0) 934 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.771635+0000 mon.a (mon.0) 935 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.771845+0000 mon.a (mon.0) 936 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.771951+0000 mon.a (mon.0) 937 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772104+0000 mon.a (mon.0) 938 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772211+0000 mon.a (mon.0) 939 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772356+0000 mon.a (mon.0) 940 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772512+0000 mon.a (mon.0) 941 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772643+0000 mon.a (mon.0) 942 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772807+0000 mon.a (mon.0) 943 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.772926+0000 mon.a (mon.0) 944 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.773074+0000 mon.a (mon.0) 945 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.773187+0000 mon.a (mon.0) 946 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.171947+0000 mon.a (mon.0) 947 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.172079+0000 mon.a (mon.0) 948 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.172215+0000 mon.a (mon.0) 949 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.172359+0000 mon.a (mon.0) 950 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.172481+0000 mon.a (mon.0) 951 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.771185+0000 mon.a (mon.0) 932 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.771220+0000 mon.a (mon.0) 933 : cluster [INF] osd.7 failed (root=default,host=smithi204) (connection refused reported by osd.3) 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.771493+0000 mon.a (mon.0) 934 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.771635+0000 mon.a (mon.0) 935 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.771845+0000 mon.a (mon.0) 936 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.771951+0000 mon.a (mon.0) 937 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772104+0000 mon.a (mon.0) 938 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772211+0000 mon.a (mon.0) 939 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772356+0000 mon.a (mon.0) 940 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772512+0000 mon.a (mon.0) 941 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772643+0000 mon.a (mon.0) 942 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772807+0000 mon.a (mon.0) 943 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.772926+0000 mon.a (mon.0) 944 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.773074+0000 mon.a (mon.0) 945 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.773187+0000 mon.a (mon.0) 946 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.171947+0000 mon.a (mon.0) 947 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.172079+0000 mon.a (mon.0) 948 : cluster [DBG] osd.7 reported immediately failed by osd.3 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.172215+0000 mon.a (mon.0) 949 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.172359+0000 mon.a (mon.0) 950 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.172481+0000 mon.a (mon.0) 951 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.172679+0000 mon.a (mon.0) 952 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.172918+0000 mon.a (mon.0) 953 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.173045+0000 mon.a (mon.0) 954 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.173215+0000 mon.a (mon.0) 955 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.173337+0000 mon.a (mon.0) 956 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.173511+0000 mon.a (mon.0) 957 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.173682+0000 mon.a (mon.0) 958 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.173858+0000 mon.a (mon.0) 959 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.174027+0000 mon.a (mon.0) 960 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.451748+0000 mon.a (mon.0) 961 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.453633+0000 mon.a (mon.0) 962 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.454747+0000 mon.a (mon.0) 963 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.462191+0000 mon.a (mon.0) 964 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.943 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.172679+0000 mon.a (mon.0) 952 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.172918+0000 mon.a (mon.0) 953 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.173045+0000 mon.a (mon.0) 954 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.173215+0000 mon.a (mon.0) 955 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.173337+0000 mon.a (mon.0) 956 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.173511+0000 mon.a (mon.0) 957 : cluster [DBG] osd.7 reported immediately failed by osd.6 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.173682+0000 mon.a (mon.0) 958 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.173858+0000 mon.a (mon.0) 959 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.174027+0000 mon.a (mon.0) 960 : cluster [DBG] osd.7 reported immediately failed by osd.4 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.451748+0000 mon.a (mon.0) 961 : cluster [DBG] osd.7 reported immediately failed by osd.2 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.453633+0000 mon.a (mon.0) 962 : cluster [DBG] osd.7 reported immediately failed by osd.5 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.454747+0000 mon.a (mon.0) 963 : cluster [DBG] osd.7 reported immediately failed by osd.1 2024-02-10T09:51:22.944 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:22 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.462191+0000 mon.a (mon.0) 964 : cluster [DBG] osd.7 reported immediately failed by osd.0 2024-02-10T09:51:23.749 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:51:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:51:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:21.995722+0000 mgr.y (mgr.25138) 317 : cluster [DBG] pgmap v195: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.590044+0000 mon.a (mon.0) 965 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:51:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.590076+0000 mon.a (mon.0) 966 : cluster [WRN] Health check failed: all OSDs are running pacific or later but require_osd_release < pacific (OSD_UPGRADE_FINISHED) 2024-02-10T09:51:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:22.600439+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e109: 8 total, 7 up, 8 in 2024-02-10T09:51:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:23.081591+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:21.995722+0000 mgr.y (mgr.25138) 317 : cluster [DBG] pgmap v195: 129 pgs: 129 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.590044+0000 mon.a (mon.0) 965 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T09:51:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.590076+0000 mon.a (mon.0) 966 : cluster [WRN] Health check failed: all OSDs are running pacific or later but require_osd_release < pacific (OSD_UPGRADE_FINISHED) 2024-02-10T09:51:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:22.600439+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e109: 8 total, 7 up, 8 in 2024-02-10T09:51:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:23.081591+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:24.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:23 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.7.service: Succeeded. 2024-02-10T09:51:24.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:23 smithi204 systemd[1]: Stopped Ceph osd.7 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:51:24.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:23 smithi204 systemd[1]: Starting Ceph osd.7 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:51:24.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:23.599883+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e110: 8 total, 7 up, 8 in 2024-02-10T09:51:24.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:23.599883+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e110: 8 total, 7 up, 8 in 2024-02-10T09:51:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:24] "GET /metrics HTTP/1.1" 200 166477 "" "Prometheus/2.33.4" 2024-02-10T09:51:25.331 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T09:51:25.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:25 smithi204 bash[68026]: --> ceph-volume lvm activate successful for osd ID: 7 2024-02-10T09:51:25.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:23.996470+0000 mgr.y (mgr.25138) 318 : cluster [DBG] pgmap v198: 129 pgs: 11 peering, 10 stale+active+clean, 108 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:25.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:24.603264+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-10T09:51:25.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:23.996470+0000 mgr.y (mgr.25138) 318 : cluster [DBG] pgmap v198: 129 pgs: 11 peering, 10 stale+active+clean, 108 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail 2024-02-10T09:51:25.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:24.603264+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-10T09:51:26.331 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:26 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:51:26.123+0000 7f7ac837f700 1 -- 172.21.15.204:0/63087022 <== mon.2 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560613c1c680 con 0x560613be3800 2024-02-10T09:51:26.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:26 smithi204 bash[68294]: 1832086c6649d2ccfb1da24023e3c30d548ecceb0a03e772466d9e11f80050f5 2024-02-10T09:51:26.332 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:26 smithi204 systemd[1]: Started Ceph osd.7 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:51:27.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:26 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:51:26.645+0000 7f7acfe0d380 -1 Falling back to public interface 2024-02-10T09:51:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:25.997095+0000 mgr.y (mgr.25138) 319 : cluster [DBG] pgmap v199: 129 pgs: 9 active+undersized+degraded, 11 peering, 8 stale+active+clean, 6 active+undersized, 95 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 23/609 objects degraded (3.777%) 2024-02-10T09:51:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:26.255821+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:26.256893+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:26.258692+0000 mon.a (mon.0) 973 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:51:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:26.612297+0000 mon.a (mon.0) 974 : cluster [WRN] Health check failed: Degraded data redundancy: 23/609 objects degraded (3.777%), 9 pgs degraded (PG_DEGRADED) 2024-02-10T09:51:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:25.997095+0000 mgr.y (mgr.25138) 319 : cluster [DBG] pgmap v199: 129 pgs: 9 active+undersized+degraded, 11 peering, 8 stale+active+clean, 6 active+undersized, 95 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 23/609 objects degraded (3.777%) 2024-02-10T09:51:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:26.255821+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:26.256893+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:26.258692+0000 mon.a (mon.0) 973 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:51:27.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:26.612297+0000 mon.a (mon.0) 974 : cluster [WRN] Health check failed: Degraded data redundancy: 23/609 objects degraded (3.777%), 9 pgs degraded (PG_DEGRADED) 2024-02-10T09:51:28.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:28.255086+0000 mon.a (mon.0) 975 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 6 pgs peering) 2024-02-10T09:51:28.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:28.255086+0000 mon.a (mon.0) 975 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 6 pgs peering) 2024-02-10T09:51:29.562 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:29 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:51:29.415+0000 7f7acfe0d380 -1 osd.7 108 log_to_monitors {default=true} 2024-02-10T09:51:29.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:27.997794+0000 mgr.y (mgr.25138) 320 : cluster [DBG] pgmap v200: 129 pgs: 26 active+undersized+degraded, 31 active+undersized, 72 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 81/609 objects degraded (13.300%) 2024-02-10T09:51:29.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:27.997794+0000 mgr.y (mgr.25138) 320 : cluster [DBG] pgmap v200: 129 pgs: 26 active+undersized+degraded, 31 active+undersized, 72 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 81/609 objects degraded (13.300%) 2024-02-10T09:51:29.907 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:51:30.559 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:51:30.559 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (19m) 115s ago 20m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:51:30.559 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (7m) 0s ago 22m 57.8M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:51:30.559 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (7m) 0s ago 32m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:51:30.559 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (15m) 115s ago 34m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (5m) 115s ago 34m 84.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (6m) 0s ago 33m 100M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (5m) 115s ago 33m 69.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (7m) 115s ago 25m 21.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (7m) 0s ago 25m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (4m) 115s ago 32m 40.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (3m) 115s ago 32m 40.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 115s ago 31m 40.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (119s) 115s ago 31m 10.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (91s) 0s ago 31m 41.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (62s) 0s ago 31m 42.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (33s) 0s ago 30m 40.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:51:30.560 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (4s) 0s ago 30m 13.5M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:51:30.561 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (19m) 0s ago 19m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:51:30.561 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (22m) 115s ago 22m 39.1M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:51:30.561 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (22m) 0s ago 22m 39.7M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:51:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:29.420008+0000 mon.b (mon.2) 13 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/2643331877,v1:172.21.15.204:6825/2643331877]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:51:30.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:29.420428+0000 mon.a (mon.0) 976 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:51:30.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:29.420008+0000 mon.b (mon.2) 13 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/2643331877,v1:172.21.15.204:6825/2643331877]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:51:30.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:30 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:29.420428+0000 mon.a (mon.0) 976 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T09:51:30.831 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:51:30 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:51:30.342+0000 7f7ac0af3700 -1 osd.7 108 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:51:30.935 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 7 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2, 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 12 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:51:30.936 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout: "osd" 2024-02-10T09:51:31.252 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-02-10T09:51:31.253 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "8/8 daemons upgraded", 2024-02-10T09:51:31.253 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T09:51:31.253 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:51:31.253 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:51:31.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:29.896315+0000 mgr.y (mgr.25138) 321 : audit [DBG] from='client.54504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:31.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:29.998264+0000 mgr.y (mgr.25138) 322 : cluster [DBG] pgmap v201: 129 pgs: 26 active+undersized+degraded, 31 active+undersized, 72 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 81/609 objects degraded (13.300%) 2024-02-10T09:51:31.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.210081+0000 mgr.y (mgr.25138) 323 : audit [DBG] from='client.64459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:31.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.316993+0000 mon.a (mon.0) 977 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T09:51:31.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:30.317056+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e111: 8 total, 7 up, 8 in 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.321612+0000 mon.b (mon.2) 14 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/2643331877,v1:172.21.15.204:6825/2643331877]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.321978+0000 mon.a (mon.0) 979 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.522476+0000 mon.a (mon.0) 980 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.523306+0000 mon.a (mon.0) 981 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "container_image"}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.547581+0000 mgr.y (mgr.25138) 324 : audit [DBG] from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.549335+0000 mon.a (mon.0) 982 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.549725+0000 mon.a (mon.0) 983 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.550032+0000 mon.a (mon.0) 984 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.550321+0000 mon.a (mon.0) 985 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:29.896315+0000 mgr.y (mgr.25138) 321 : audit [DBG] from='client.54504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:31.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:29.998264+0000 mgr.y (mgr.25138) 322 : cluster [DBG] pgmap v201: 129 pgs: 26 active+undersized+degraded, 31 active+undersized, 72 active+clean; 7.0 KiB data, 2.0 GiB used, 712 GiB / 715 GiB avail; 81/609 objects degraded (13.300%) 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.210081+0000 mgr.y (mgr.25138) 323 : audit [DBG] from='client.64459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.316993+0000 mon.a (mon.0) 977 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:30.317056+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e111: 8 total, 7 up, 8 in 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.321612+0000 mon.b (mon.2) 14 : audit [INF] from='osd.7 [v2:172.21.15.204:6824/2643331877,v1:172.21.15.204:6825/2643331877]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.321978+0000 mon.a (mon.0) 979 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.522476+0000 mon.a (mon.0) 980 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.523306+0000 mon.a (mon.0) 981 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd","key": "container_image"}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.547581+0000 mgr.y (mgr.25138) 324 : audit [DBG] from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.549335+0000 mon.a (mon.0) 982 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.549725+0000 mon.a (mon.0) 983 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.550032+0000 mon.a (mon.0) 984 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.550321+0000 mon.a (mon.0) 985 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.550629+0000 mon.a (mon.0) 986 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.550928+0000 mon.a (mon.0) 987 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.552569+0000 mon.a (mon.0) 988 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.552877+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.553194+0000 mon.a (mon.0) 990 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.553475+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.553794+0000 mon.a (mon.0) 992 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:30.934704+0000 mon.c (mon.1) 14 : audit [DBG] from='client.54522 172.21.15.153:0/420497966' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.550629+0000 mon.a (mon.0) 986 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.550928+0000 mon.a (mon.0) 987 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.552569+0000 mon.a (mon.0) 988 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:51:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.552877+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.553194+0000 mon.a (mon.0) 990 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.553475+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.553794+0000 mon.a (mon.0) 992 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:51:31.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:30.934704+0000 mon.c (mon.1) 14 : audit [DBG] from='client.54522 172.21.15.153:0/420497966' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:31.251591+0000 mgr.y (mgr.25138) 325 : audit [DBG] from='client.64477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:31.314278+0000 mon.a (mon.0) 993 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:51:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:31.318637+0000 mon.a (mon.0) 994 : cluster [INF] osd.7 [v2:172.21.15.204:6824/2643331877,v1:172.21.15.204:6825/2643331877] boot 2024-02-10T09:51:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:31.318683+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-10T09:51:32.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:31.321904+0000 mon.a (mon.0) 996 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:32.321596+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:31.251591+0000 mgr.y (mgr.25138) 325 : audit [DBG] from='client.64477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:31.314278+0000 mon.a (mon.0) 993 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:31.318637+0000 mon.a (mon.0) 994 : cluster [INF] osd.7 [v2:172.21.15.204:6824/2643331877,v1:172.21.15.204:6825/2643331877] boot 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:31.318683+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:31.321904+0000 mon.a (mon.0) 996 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T09:51:32.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:32.321596+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-10T09:51:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:31.998798+0000 mgr.y (mgr.25138) 326 : cluster [DBG] pgmap v204: 129 pgs: 16 peering, 23 active+undersized+degraded, 18 active+undersized, 72 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 74/609 objects degraded (12.151%) 2024-02-10T09:51:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:32.330658+0000 mon.a (mon.0) 998 : cluster [WRN] Health check update: Degraded data redundancy: 74/609 objects degraded (12.151%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T09:51:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:31.998798+0000 mgr.y (mgr.25138) 326 : cluster [DBG] pgmap v204: 129 pgs: 16 peering, 23 active+undersized+degraded, 18 active+undersized, 72 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 74/609 objects degraded (12.151%) 2024-02-10T09:51:33.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:32.330658+0000 mon.a (mon.0) 998 : cluster [WRN] Health check update: Degraded data redundancy: 74/609 objects degraded (12.151%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T09:51:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:51:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:51:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:34] "GET /metrics HTTP/1.1" 200 166477 "" "Prometheus/2.33.4" 2024-02-10T09:51:34.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:33.666692+0000 mon.a (mon.0) 999 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:51:34.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:33.885860+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.172006+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.177205+0000 mon.a (mon.0) 1002 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.179653+0000 mon.a (mon.0) 1003 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.181038+0000 mon.a (mon.0) 1004 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.182187+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.184066+0000 mon.a (mon.0) 1006 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.191298+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.192179+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.196385+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.197142+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.200774+0000 mon.a (mon.0) 1011 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.201530+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T09:51:34.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.205707+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:33.666692+0000 mon.a (mon.0) 999 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:33.885860+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.172006+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.177205+0000 mon.a (mon.0) 1002 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.179653+0000 mon.a (mon.0) 1003 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.181038+0000 mon.a (mon.0) 1004 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.182187+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.184066+0000 mon.a (mon.0) 1006 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.191298+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.192179+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.196385+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.197142+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.200774+0000 mon.a (mon.0) 1011 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T09:51:34.691 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.201530+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.205707+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.206441+0000 mon.a (mon.0) 1014 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.210538+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.211253+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.215028+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.216238+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.236210+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.237407+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.242314+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T09:51:34.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.243374+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.250249+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:34.252117+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "pacific"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.206441+0000 mon.a (mon.0) 1014 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.210538+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.211253+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.215028+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.216238+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.236210+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.237407+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.242314+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.243374+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.250249+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T09:51:34.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:34.252117+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "pacific"}]: dispatch 2024-02-10T09:51:35.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:33.999449+0000 mgr.y (mgr.25138) 327 : cluster [DBG] pgmap v206: 129 pgs: 16 peering, 17 active+undersized+degraded, 13 active+undersized, 83 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57/609 objects degraded (9.360%) 2024-02-10T09:51:35.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:34.184704+0000 mgr.y (mgr.25138) 328 : cephadm [INF] Upgrade: Setting container_image for all osd 2024-02-10T09:51:35.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:34.251681+0000 mgr.y (mgr.25138) 329 : cephadm [INF] Upgrade: Setting require_osd_release to 16 pacific 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:35.246790+0000 mon.a (mon.0) 1025 : cluster [INF] Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running pacific or later but require_osd_release < pacific) 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.251459+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "pacific"}]': finished 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:35.251521+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.254754+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.260416+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.262321+0000 mon.a (mon.0) 1030 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.267528+0000 mon.a (mon.0) 1031 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.268656+0000 mon.a (mon.0) 1032 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.269668+0000 mon.a (mon.0) 1033 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.273755+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.274803+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.278824+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.280067+0000 mon.a (mon.0) 1037 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:33.999449+0000 mgr.y (mgr.25138) 327 : cluster [DBG] pgmap v206: 129 pgs: 16 peering, 17 active+undersized+degraded, 13 active+undersized, 83 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57/609 objects degraded (9.360%) 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:34.184704+0000 mgr.y (mgr.25138) 328 : cephadm [INF] Upgrade: Setting container_image for all osd 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:34.251681+0000 mgr.y (mgr.25138) 329 : cephadm [INF] Upgrade: Setting require_osd_release to 16 pacific 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:35.246790+0000 mon.a (mon.0) 1025 : cluster [INF] Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running pacific or later but require_osd_release < pacific) 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.251459+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "pacific"}]': finished 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:35.251521+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.254754+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.260416+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.262321+0000 mon.a (mon.0) 1030 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.267528+0000 mon.a (mon.0) 1031 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.268656+0000 mon.a (mon.0) 1032 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.269668+0000 mon.a (mon.0) 1033 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.273755+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.274803+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.278824+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.280067+0000 mon.a (mon.0) 1037 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.281201+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.282291+0000 mon.a (mon.0) 1039 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.283397+0000 mon.a (mon.0) 1040 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.284664+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.287927+0000 mon.a (mon.0) 1042 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.288616+0000 mon.a (mon.0) 1043 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.291676+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.292423+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:51:35.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.295405+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.296040+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.298867+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.299534+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.302224+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.302884+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.281201+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.282291+0000 mon.a (mon.0) 1039 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.283397+0000 mon.a (mon.0) 1040 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.284664+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.287927+0000 mon.a (mon.0) 1042 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.288616+0000 mon.a (mon.0) 1043 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.693 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.291676+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.292423+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.295405+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.296040+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.298867+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.299534+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.302224+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.302884+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.303458+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.306231+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:51:35.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.306820+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.307375+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.312487+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.313138+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.318384+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.319014+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.319548+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.320042+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.320536+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.321032+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.324538+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.325401+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.328100+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.328499+0000 mon.a (mon.0) 1067 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.333132+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.695 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.336897+0000 mon.a (mon.0) 1069 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:35.341539+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.303458+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.306231+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.306820+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.307375+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.312487+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.313138+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.318384+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:51:35.696 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.319014+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.319548+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.320042+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.320536+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.321032+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.324538+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.325401+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.328100+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.328499+0000 mon.a (mon.0) 1067 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.333132+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.336897+0000 mon.a (mon.0) 1069 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:51:35.697 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:35.341539+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:35.255616+0000 mgr.y (mgr.25138) 330 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:51:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:35.263471+0000 mgr.y (mgr.25138) 331 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:51:36.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:35.270260+0000 mgr.y (mgr.25138) 332 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:35.275383+0000 mgr.y (mgr.25138) 333 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:35.283985+0000 mgr.y (mgr.25138) 334 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:51:35.325014+0000 mgr.y (mgr.25138) 335 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:35.255616+0000 mgr.y (mgr.25138) 330 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:35.263471+0000 mgr.y (mgr.25138) 331 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:35.270260+0000 mgr.y (mgr.25138) 332 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:35.275383+0000 mgr.y (mgr.25138) 333 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:35.283985+0000 mgr.y (mgr.25138) 334 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:51:36.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:51:35.325014+0000 mgr.y (mgr.25138) 335 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:51:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:36.000183+0000 mgr.y (mgr.25138) 336 : cluster [DBG] pgmap v208: 129 pgs: 16 peering, 113 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:36.413881+0000 mon.a (mon.0) 1071 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/609 objects degraded (9.360%), 17 pgs degraded) 2024-02-10T09:51:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:36.413918+0000 mon.a (mon.0) 1072 : cluster [INF] Cluster is now healthy 2024-02-10T09:51:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:36.000183+0000 mgr.y (mgr.25138) 336 : cluster [DBG] pgmap v208: 129 pgs: 16 peering, 113 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:36.413881+0000 mon.a (mon.0) 1071 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/609 objects degraded (9.360%), 17 pgs degraded) 2024-02-10T09:51:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:36.413918+0000 mon.a (mon.0) 1072 : cluster [INF] Cluster is now healthy 2024-02-10T09:51:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:38.000767+0000 mgr.y (mgr.25138) 337 : cluster [DBG] pgmap v209: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:38.081130+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:38.302772+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:38.000767+0000 mgr.y (mgr.25138) 337 : cluster [DBG] pgmap v209: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:38.081130+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:39.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:38.302772+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:41.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:40.001317+0000 mgr.y (mgr.25138) 338 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:41.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:40.001317+0000 mgr.y (mgr.25138) 338 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:43.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:42.001994+0000 mgr.y (mgr.25138) 339 : cluster [DBG] pgmap v211: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:43.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:42.001994+0000 mgr.y (mgr.25138) 339 : cluster [DBG] pgmap v211: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:44.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:51:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:51:44.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:51:43.309685+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:44.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:51:43.309685+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:51:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:44] "GET /metrics HTTP/1.1" 200 167232 "" "Prometheus/2.33.4" 2024-02-10T09:51:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:44.002554+0000 mgr.y (mgr.25138) 340 : cluster [DBG] pgmap v212: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:45.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:44.002554+0000 mgr.y (mgr.25138) 340 : cluster [DBG] pgmap v212: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:46.003257+0000 mgr.y (mgr.25138) 341 : cluster [DBG] pgmap v213: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:46.003257+0000 mgr.y (mgr.25138) 341 : cluster [DBG] pgmap v213: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:49.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:48.004013+0000 mgr.y (mgr.25138) 342 : cluster [DBG] pgmap v214: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:49.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:48.004013+0000 mgr.y (mgr.25138) 342 : cluster [DBG] pgmap v214: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:51.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:50.004528+0000 mgr.y (mgr.25138) 343 : cluster [DBG] pgmap v215: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:51.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:50.004528+0000 mgr.y (mgr.25138) 343 : cluster [DBG] pgmap v215: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:53.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:52.005256+0000 mgr.y (mgr.25138) 344 : cluster [DBG] pgmap v216: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:53.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:52.005256+0000 mgr.y (mgr.25138) 344 : cluster [DBG] pgmap v216: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:51:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:51:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:51:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:51:54] "GET /metrics HTTP/1.1" 200 167242 "" "Prometheus/2.33.4" 2024-02-10T09:51:55.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:54.005998+0000 mgr.y (mgr.25138) 345 : cluster [DBG] pgmap v217: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:55.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:54.005998+0000 mgr.y (mgr.25138) 345 : cluster [DBG] pgmap v217: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:56.006811+0000 mgr.y (mgr.25138) 346 : cluster [DBG] pgmap v218: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:57.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:56.006811+0000 mgr.y (mgr.25138) 346 : cluster [DBG] pgmap v218: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:59.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:51:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:51:58.007411+0000 mgr.y (mgr.25138) 347 : cluster [DBG] pgmap v219: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:51:59.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:51:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:51:58.007411+0000 mgr.y (mgr.25138) 347 : cluster [DBG] pgmap v219: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:01.613 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:00.008032+0000 mgr.y (mgr.25138) 348 : cluster [DBG] pgmap v220: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:01.614 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:00.008032+0000 mgr.y (mgr.25138) 348 : cluster [DBG] pgmap v220: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:02.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:52:02.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:01.579127+0000 mgr.y (mgr.25138) 349 : audit [DBG] from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:02.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:02 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:01.579127+0000 mgr.y (mgr.25138) 349 : audit [DBG] from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:03.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:02.008731+0000 mgr.y (mgr.25138) 350 : cluster [DBG] pgmap v221: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:03.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:02.008731+0000 mgr.y (mgr.25138) 350 : cluster [DBG] pgmap v221: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:52:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:52:04.412 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:52:04.412 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (20m) 2m ago 20m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:52:04.412 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (7m) 33s ago 23m 57.8M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:52:04.413 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (8m) 33s ago 33m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:52:04.413 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (16m) 2m ago 35m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:52:04.413 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (6m) 2m ago 35m 84.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:52:04.413 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (6m) 33s ago 33m 100M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:52:04.413 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (5m) 2m ago 34m 69.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:52:04.413 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (8m) 2m ago 25m 21.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:52:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (8m) 33s ago 25m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:52:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (4m) 2m ago 33m 40.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:52:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 2m ago 32m 40.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:52:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (3m) 2m ago 32m 40.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:52:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (2m) 2m ago 32m 10.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (2m) 33s ago 31m 41.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (96s) 33s ago 31m 42.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (67s) 33s ago 31m 40.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (38s) 33s ago 30m 13.5M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (20m) 33s ago 20m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (23m) 2m ago 23m 39.1M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:52:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (23m) 33s ago 23m 39.7M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:52:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:04] "GET /metrics HTTP/1.1" 200 167242 "" "Prometheus/2.33.4" 2024-02-10T09:52:05.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-02-10T09:52:05.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:04.009429+0000 mgr.y (mgr.25138) 351 : cluster [DBG] pgmap v222: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:05.621 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.400472+0000 mgr.y (mgr.25138) 352 : audit [DBG] from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.402193+0000 mon.a (mon.0) 1076 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.402545+0000 mon.a (mon.0) 1077 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.402824+0000 mon.a (mon.0) 1078 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.403081+0000 mon.a (mon.0) 1079 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.403316+0000 mon.a (mon.0) 1080 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.403554+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.405127+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.405381+0000 mon.a (mon.0) 1083 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.405627+0000 mon.a (mon.0) 1084 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.405875+0000 mon.a (mon.0) 1085 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.622 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:04.406109+0000 mon.a (mon.0) 1086 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:04.009429+0000 mgr.y (mgr.25138) 351 : cluster [DBG] pgmap v222: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.400472+0000 mgr.y (mgr.25138) 352 : audit [DBG] from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.402193+0000 mon.a (mon.0) 1076 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.402545+0000 mon.a (mon.0) 1077 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.402824+0000 mon.a (mon.0) 1078 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.403081+0000 mon.a (mon.0) 1079 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.403316+0000 mon.a (mon.0) 1080 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.403554+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.405127+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.405381+0000 mon.a (mon.0) 1083 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.405627+0000 mon.a (mon.0) 1084 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.405875+0000 mon.a (mon.0) 1085 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:05.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:04.406109+0000 mon.a (mon.0) 1086 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:07.162 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:52:07.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:06.010100+0000 mgr.y (mgr.25138) 353 : cluster [DBG] pgmap v223: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:07.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:07.150829+0000 mon.a (mon.0) 1087 : audit [DBG] from='client.54546 172.21.15.153:0/2156385843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:07.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:06.010100+0000 mgr.y (mgr.25138) 353 : cluster [DBG] pgmap v223: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:07.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:07.150829+0000 mon.a (mon.0) 1087 : audit [DBG] from='client.54546 172.21.15.153:0/2156385843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:07.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-02-10T09:52:08.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:07.937327+0000 mon.a (mon.0) 1088 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:07.937578+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:07.937793+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:07.937990+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:07.937327+0000 mon.a (mon.0) 1088 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:07.937578+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:07.937793+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T09:52:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:07.937990+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-10T09:52:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:08.010597+0000 mgr.y (mgr.25138) 354 : cluster [DBG] pgmap v224: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:09.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:08.010597+0000 mgr.y (mgr.25138) 354 : cluster [DBG] pgmap v224: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:09.834 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)" 2024-02-10T09:52:10.618 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:09.823001+0000 mon.c (mon.1) 15 : audit [DBG] from='client.54552 172.21.15.153:0/3146671100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:10.619 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:10 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:09.823001+0000 mon.c (mon.1) 15 : audit [DBG] from='client.54552 172.21.15.153:0/3146671100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:10.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.r.z' 2024-02-10T09:52:11.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:10.010975+0000 mgr.y (mgr.25138) 355 : cluster [DBG] pgmap v225: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:11.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:10.010975+0000 mgr.y (mgr.25138) 355 : cluster [DBG] pgmap v225: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:13.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:12.011621+0000 mgr.y (mgr.25138) 356 : cluster [DBG] pgmap v226: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:12.011621+0000 mgr.y (mgr.25138) 356 : cluster [DBG] pgmap v226: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:52:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:52:14.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:12.858916+0000 mgr.y (mgr.25138) 357 : audit [DBG] from='client.54558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "services": "rgw.r.z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:14] "GET /metrics HTTP/1.1" 200 167233 "" "Prometheus/2.33.4" 2024-02-10T09:52:14.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:12.858916+0000 mgr.y (mgr.25138) 357 : audit [DBG] from='client.54558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "services": "rgw.r.z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:14.837 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:52:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:14.012071+0000 mgr.y (mgr.25138) 358 : cluster [DBG] pgmap v227: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:14.836837+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:14.850975+0000 mon.a (mon.0) 1093 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:15.231490+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:14.012071+0000 mgr.y (mgr.25138) 358 : cluster [DBG] pgmap v227: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:14.836837+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:14.850975+0000 mon.a (mon.0) 1093 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:15.558 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:15.231490+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:15.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:52:16.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:14.831052+0000 mgr.y (mgr.25138) 359 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:52:16.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:15.234726+0000 mgr.y (mgr.25138) 360 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:52:16.635 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:14.831052+0000 mgr.y (mgr.25138) 359 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:52:16.636 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:16 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:15.234726+0000 mgr.y (mgr.25138) 360 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:52:17.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:16.012556+0000 mgr.y (mgr.25138) 361 : cluster [DBG] pgmap v228: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:17.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:16.012556+0000 mgr.y (mgr.25138) 361 : cluster [DBG] pgmap v228: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:17.742 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (20m) 2m ago 21m 26.4M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (8m) 47s ago 23m 57.8M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (8m) 47s ago 33m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (16m) 2m ago 35m 434M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (6m) 2m ago 35m 84.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (7m) 47s ago 33m 100M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (6m) 2m ago 34m 69.9M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (8m) 2m ago 26m 21.5M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (8m) 47s ago 26m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (4m) 2m ago 33m 40.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:52:18.392 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 2m ago 33m 40.7M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (3m) 2m ago 32m 40.8M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (2m) 2m ago 32m 10.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (2m) 47s ago 32m 41.1M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (110s) 47s ago 31m 42.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (81s) 47s ago 31m 40.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (52s) 47s ago 31m 13.5M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (20m) 47s ago 20m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (23m) 2m ago 23m 39.1M - 15.2.0 204a01f9b0b6 4417ab8bf66a 2024-02-10T09:52:18.393 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (23m) 47s ago 23m 39.7M - 15.2.0 204a01f9b0b6 a9860a65eacb 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.382094+0000 mon.a (mon.0) 1095 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.382411+0000 mon.a (mon.0) 1096 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.382644+0000 mon.a (mon.0) 1097 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.382894+0000 mon.a (mon.0) 1098 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.383126+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.383358+0000 mon.a (mon.0) 1100 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.384860+0000 mon.a (mon.0) 1101 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.385119+0000 mon.a (mon.0) 1102 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.385355+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.399 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.385620+0000 mon.a (mon.0) 1104 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.386089+0000 mon.a (mon.0) 1105 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.382094+0000 mon.a (mon.0) 1095 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:18.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.382411+0000 mon.a (mon.0) 1096 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.382644+0000 mon.a (mon.0) 1097 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.382894+0000 mon.a (mon.0) 1098 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.383126+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.383358+0000 mon.a (mon.0) 1100 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.384860+0000 mon.a (mon.0) 1101 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.385119+0000 mon.a (mon.0) 1102 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.385355+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.385620+0000 mon.a (mon.0) 1104 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.386089+0000 mon.a (mon.0) 1105 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 8 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:52:18.776 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2, 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 13 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:52:18.777 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:52:19.123 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:52:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons in service(s) rgw.r.z", 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading rgw daemons", 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:52:19.124 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:52:19.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:17.729997+0000 mgr.y (mgr.25138) 362 : audit [DBG] from='client.54564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:19.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:18.013102+0000 mgr.y (mgr.25138) 363 : cluster [DBG] pgmap v229: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.062984+0000 mgr.y (mgr.25138) 364 : audit [DBG] from='client.54570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.380256+0000 mgr.y (mgr.25138) 365 : audit [DBG] from='client.64525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.775357+0000 mon.c (mon.1) 16 : audit [DBG] from='client.64531 172.21.15.153:0/3427483613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.832006+0000 mon.a (mon.0) 1106 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.833310+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.838317+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.839638+0000 mon.a (mon.0) 1109 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.844168+0000 mon.a (mon.0) 1110 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.845465+0000 mon.a (mon.0) 1111 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.849816+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.850838+0000 mon.a (mon.0) 1113 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.855150+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.656 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.856457+0000 mon.a (mon.0) 1115 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.657 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.861108+0000 mon.a (mon.0) 1116 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.657 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.862691+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.657 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:18.867094+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.657 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:17.729997+0000 mgr.y (mgr.25138) 362 : audit [DBG] from='client.54564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:19.657 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:18.013102+0000 mgr.y (mgr.25138) 363 : cluster [DBG] pgmap v229: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.062984+0000 mgr.y (mgr.25138) 364 : audit [DBG] from='client.54570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.380256+0000 mgr.y (mgr.25138) 365 : audit [DBG] from='client.64525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.775357+0000 mon.c (mon.1) 16 : audit [DBG] from='client.64531 172.21.15.153:0/3427483613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.832006+0000 mon.a (mon.0) 1106 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.833310+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.838317+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.839638+0000 mon.a (mon.0) 1109 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.844168+0000 mon.a (mon.0) 1110 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.845465+0000 mon.a (mon.0) 1111 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.849816+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.658 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.850838+0000 mon.a (mon.0) 1113 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.659 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.855150+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.659 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.856457+0000 mon.a (mon.0) 1115 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.659 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.861108+0000 mon.a (mon.0) 1116 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:19.659 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.862691+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:19.659 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:18.867094+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:20.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.832367+0000 mgr.y (mgr.25138) 366 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:52:20.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.832424+0000 mgr.y (mgr.25138) 367 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:52:20.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.840189+0000 mgr.y (mgr.25138) 368 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.846021+0000 mgr.y (mgr.25138) 369 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.851378+0000 mgr.y (mgr.25138) 370 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.857124+0000 mgr.y (mgr.25138) 371 : cephadm [INF] Upgrade: Setting container_image for all osd 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:18.863296+0000 mgr.y (mgr.25138) 372 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:19.122828+0000 mgr.y (mgr.25138) 373 : audit [DBG] from='client.54588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:20.330009+0000 mon.a (mon.0) 1119 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:20.330688+0000 mon.a (mon.0) 1120 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:20.331819+0000 mon.a (mon.0) 1121 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:20.332620+0000 mon.a (mon.0) 1122 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi153.zyveqt","key": "container_image"}]: dispatch 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.832367+0000 mgr.y (mgr.25138) 366 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.832424+0000 mgr.y (mgr.25138) 367 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:52:20.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.840189+0000 mgr.y (mgr.25138) 368 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.846021+0000 mgr.y (mgr.25138) 369 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.851378+0000 mgr.y (mgr.25138) 370 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.857124+0000 mgr.y (mgr.25138) 371 : cephadm [INF] Upgrade: Setting container_image for all osd 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:18.863296+0000 mgr.y (mgr.25138) 372 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:19.122828+0000 mgr.y (mgr.25138) 373 : audit [DBG] from='client.54588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:20.330009+0000 mon.a (mon.0) 1119 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:20.330688+0000 mon.a (mon.0) 1120 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi153.zyveqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:20.331819+0000 mon.a (mon.0) 1121 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:52:20.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:20 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:20.332620+0000 mon.a (mon.0) 1122 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi153.zyveqt","key": "container_image"}]: dispatch 2024-02-10T09:52:21.639 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:20.013577+0000 mgr.y (mgr.25138) 374 : cluster [DBG] pgmap v230: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:21.639 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:20.324866+0000 mgr.y (mgr.25138) 375 : cephadm [INF] Upgrade: Updating rgw.r.z.smithi153.zyveqt (1/2) 2024-02-10T09:52:21.639 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:20.332292+0000 mgr.y (mgr.25138) 376 : cephadm [INF] Deploying daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:52:21.639 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:20.013577+0000 mgr.y (mgr.25138) 374 : cluster [DBG] pgmap v230: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:21.640 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:20.324866+0000 mgr.y (mgr.25138) 375 : cephadm [INF] Upgrade: Updating rgw.r.z.smithi153.zyveqt (1/2) 2024-02-10T09:52:21.640 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:20.332292+0000 mgr.y (mgr.25138) 376 : cephadm [INF] Deploying daemon rgw.r.z.smithi153.zyveqt on smithi153 2024-02-10T09:52:23.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:22.014185+0000 mgr.y (mgr.25138) 377 : cluster [DBG] pgmap v231: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:23.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:22.014185+0000 mgr.y (mgr.25138) 377 : cluster [DBG] pgmap v231: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:52:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:52:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:24] "GET /metrics HTTP/1.1" 200 164571 "" "Prometheus/2.33.4" 2024-02-10T09:52:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:23.924773+0000 mon.a (mon.0) 1123 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:25.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:24.014598+0000 mgr.y (mgr.25138) 378 : cluster [DBG] pgmap v232: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:23.924773+0000 mon.a (mon.0) 1123 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:25.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:24.014598+0000 mgr.y (mgr.25138) 378 : cluster [DBG] pgmap v232: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:25.422196+0000 mgr.y (mgr.25138) 379 : cephadm [INF] Upgrade: Updating rgw.r.z.smithi204.kmhqqg (2/2) 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:25.429851+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:25.430710+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:25.432039+0000 mon.a (mon.0) 1126 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:25.432644+0000 mgr.y (mgr.25138) 380 : cephadm [INF] Deploying daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:25.433144+0000 mon.a (mon.0) 1127 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi204.kmhqqg","key": "container_image"}]: dispatch 2024-02-10T09:52:26.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:25.422196+0000 mgr.y (mgr.25138) 379 : cephadm [INF] Upgrade: Updating rgw.r.z.smithi204.kmhqqg (2/2) 2024-02-10T09:52:26.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:25.429851+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:26.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:25.430710+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.r.z.smithi204.kmhqqg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T09:52:26.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:25.432039+0000 mon.a (mon.0) 1126 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:52:26.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:25.432644+0000 mgr.y (mgr.25138) 380 : cephadm [INF] Deploying daemon rgw.r.z.smithi204.kmhqqg on smithi204 2024-02-10T09:52:26.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:26 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:25.433144+0000 mon.a (mon.0) 1127 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "client.rgw.r.z.smithi204.kmhqqg","key": "container_image"}]: dispatch 2024-02-10T09:52:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:26.015280+0000 mgr.y (mgr.25138) 381 : cluster [DBG] pgmap v233: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 170 B/s wr, 42 op/s 2024-02-10T09:52:27.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:26.015280+0000 mgr.y (mgr.25138) 381 : cluster [DBG] pgmap v233: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 170 B/s wr, 42 op/s 2024-02-10T09:52:29.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:28.015874+0000 mgr.y (mgr.25138) 382 : cluster [DBG] pgmap v234: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 170 B/s wr, 82 op/s 2024-02-10T09:52:29.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:29.114609+0000 mon.a (mon.0) 1128 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:29.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:29.115823+0000 mon.a (mon.0) 1129 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:29.117941+0000 mon.a (mon.0) 1130 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:29.118311+0000 mon.a (mon.0) 1131 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:28.015874+0000 mgr.y (mgr.25138) 382 : cluster [DBG] pgmap v234: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 170 B/s wr, 82 op/s 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:29.114609+0000 mon.a (mon.0) 1128 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:29.115823+0000 mon.a (mon.0) 1129 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:29.117941+0000 mon.a (mon.0) 1130 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:52:29.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:29.118311+0000 mon.a (mon.0) 1131 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon","key": "container_image"}]: dispatch 2024-02-10T09:52:31.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:30.016368+0000 mgr.y (mgr.25138) 383 : cluster [DBG] pgmap v235: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 255 B/s wr, 92 op/s 2024-02-10T09:52:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:30.016368+0000 mgr.y (mgr.25138) 383 : cluster [DBG] pgmap v235: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 255 B/s wr, 92 op/s 2024-02-10T09:52:33.054 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:31.771342+0000 mon.a (mon.0) 1132 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:32.016891+0000 mgr.y (mgr.25138) 384 : cluster [DBG] pgmap v236: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 139 op/s 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.666022+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.671180+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.679044+0000 mon.a (mon.0) 1135 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.681699+0000 mon.a (mon.0) 1136 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.683188+0000 mon.a (mon.0) 1137 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.684400+0000 mon.a (mon.0) 1138 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.686222+0000 mon.a (mon.0) 1139 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.687958+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.689489+0000 mon.a (mon.0) 1141 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.695550+0000 mon.a (mon.0) 1142 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.055 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.696436+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi153.zyveqt"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.699783+0000 mon.a (mon.0) 1144 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi153.zyveqt"}]': finished 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.700504+0000 mon.a (mon.0) 1145 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi204.kmhqqg"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.704912+0000 mon.a (mon.0) 1146 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi204.kmhqqg"}]': finished 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.706069+0000 mon.a (mon.0) 1147 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.710120+0000 mon.a (mon.0) 1148 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.711244+0000 mon.a (mon.0) 1149 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.712360+0000 mon.a (mon.0) 1150 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.716367+0000 mon.a (mon.0) 1151 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.717478+0000 mon.a (mon.0) 1152 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.723837+0000 mon.a (mon.0) 1153 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.725128+0000 mon.a (mon.0) 1154 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.726316+0000 mon.a (mon.0) 1155 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.727468+0000 mon.a (mon.0) 1156 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.056 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.728589+0000 mon.a (mon.0) 1157 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.730100+0000 mon.a (mon.0) 1158 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.733339+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.733978+0000 mon.a (mon.0) 1160 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.738407+0000 mon.a (mon.0) 1161 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:31.771342+0000 mon.a (mon.0) 1132 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:32.016891+0000 mgr.y (mgr.25138) 384 : cluster [DBG] pgmap v236: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 139 op/s 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.666022+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.671180+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.679044+0000 mon.a (mon.0) 1135 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.681699+0000 mon.a (mon.0) 1136 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.683188+0000 mon.a (mon.0) 1137 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.684400+0000 mon.a (mon.0) 1138 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.686222+0000 mon.a (mon.0) 1139 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.687958+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.689489+0000 mon.a (mon.0) 1141 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.695550+0000 mon.a (mon.0) 1142 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.696436+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi153.zyveqt"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.699783+0000 mon.a (mon.0) 1144 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi153.zyveqt"}]': finished 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.700504+0000 mon.a (mon.0) 1145 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi204.kmhqqg"}]: dispatch 2024-02-10T09:52:33.058 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.739055+0000 mon.a (mon.0) 1162 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.743957+0000 mon.a (mon.0) 1163 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.744585+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.748810+0000 mon.a (mon.0) 1165 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.749469+0000 mon.a (mon.0) 1166 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.752395+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.753056+0000 mon.a (mon.0) 1168 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.755946+0000 mon.a (mon.0) 1169 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.756562+0000 mon.a (mon.0) 1170 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.759376+0000 mon.a (mon.0) 1171 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.760009+0000 mon.a (mon.0) 1172 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.760524+0000 mon.a (mon.0) 1173 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.763442+0000 mon.a (mon.0) 1174 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.764081+0000 mon.a (mon.0) 1175 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.766855+0000 mon.a (mon.0) 1176 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:52:33.059 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.767437+0000 mon.a (mon.0) 1177 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.767907+0000 mon.a (mon.0) 1178 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.768362+0000 mon.a (mon.0) 1179 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.768813+0000 mon.a (mon.0) 1180 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.704912+0000 mon.a (mon.0) 1146 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.r.z.smithi204.kmhqqg"}]': finished 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.706069+0000 mon.a (mon.0) 1147 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.710120+0000 mon.a (mon.0) 1148 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.711244+0000 mon.a (mon.0) 1149 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.712360+0000 mon.a (mon.0) 1150 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.716367+0000 mon.a (mon.0) 1151 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.717478+0000 mon.a (mon.0) 1152 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.723837+0000 mon.a (mon.0) 1153 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.725128+0000 mon.a (mon.0) 1154 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.726316+0000 mon.a (mon.0) 1155 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.727468+0000 mon.a (mon.0) 1156 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.728589+0000 mon.a (mon.0) 1157 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.730100+0000 mon.a (mon.0) 1158 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.733339+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.733978+0000 mon.a (mon.0) 1160 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.738407+0000 mon.a (mon.0) 1161 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.739055+0000 mon.a (mon.0) 1162 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.743957+0000 mon.a (mon.0) 1163 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.744585+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:52:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.748810+0000 mon.a (mon.0) 1165 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.749469+0000 mon.a (mon.0) 1166 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.752395+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.753056+0000 mon.a (mon.0) 1168 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.755946+0000 mon.a (mon.0) 1169 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.756562+0000 mon.a (mon.0) 1170 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.759376+0000 mon.a (mon.0) 1171 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.760009+0000 mon.a (mon.0) 1172 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.760524+0000 mon.a (mon.0) 1173 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.763442+0000 mon.a (mon.0) 1174 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.764081+0000 mon.a (mon.0) 1175 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.766855+0000 mon.a (mon.0) 1176 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.767437+0000 mon.a (mon.0) 1177 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.767907+0000 mon.a (mon.0) 1178 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.768362+0000 mon.a (mon.0) 1179 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:32 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.768813+0000 mon.a (mon.0) 1180 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:52:33.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STOPPING 2024-02-10T09:52:33.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:52:33.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STOPPED 2024-02-10T09:52:33.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STARTING 2024-02-10T09:52:33.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: CherryPy Checker: 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: The Application mounted at '' has an empty config. 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Serving on http://:::9283 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STARTED 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STOPPING 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STOPPED 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STARTING 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Serving on http://:::9283 2024-02-10T09:52:33.440 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:52:33] ENGINE Bus STARTED 2024-02-10T09:52:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:52:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:52:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.690144+0000 mgr.y (mgr.25138) 385 : cephadm [INF] Upgrade: Setting container_image for all rgw 2024-02-10T09:52:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.706681+0000 mgr.y (mgr.25138) 386 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:52:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.712951+0000 mgr.y (mgr.25138) 387 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:52:34.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.718093+0000 mgr.y (mgr.25138) 388 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.729238+0000 mgr.y (mgr.25138) 389 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.769257+0000 mon.a (mon.0) 1181 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.772164+0000 mon.a (mon.0) 1182 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.772870+0000 mgr.y (mgr.25138) 390 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.773271+0000 mon.a (mon.0) 1183 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.782511+0000 mon.a (mon.0) 1184 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:32.782976+0000 mon.a (mon.0) 1185 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:52:32.787255+0000 mgr.y (mgr.25138) 391 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.048947+0000 mon.a (mon.0) 1186 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.053593+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.057592+0000 mon.a (mon.0) 1188 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.064747+0000 mon.a (mon.0) 1189 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.072136+0000 mon.a (mon.0) 1190 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.075900+0000 mon.a (mon.0) 1191 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.080288+0000 mon.a (mon.0) 1192 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:33.332890+0000 mon.a (mon.0) 1193 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.690144+0000 mgr.y (mgr.25138) 385 : cephadm [INF] Upgrade: Setting container_image for all rgw 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.706681+0000 mgr.y (mgr.25138) 386 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.712951+0000 mgr.y (mgr.25138) 387 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.718093+0000 mgr.y (mgr.25138) 388 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.729238+0000 mgr.y (mgr.25138) 389 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.769257+0000 mon.a (mon.0) 1181 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.772164+0000 mon.a (mon.0) 1182 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:52:34.191 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.772870+0000 mgr.y (mgr.25138) 390 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.773271+0000 mon.a (mon.0) 1183 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.782511+0000 mon.a (mon.0) 1184 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:32.782976+0000 mon.a (mon.0) 1185 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:52:32.787255+0000 mgr.y (mgr.25138) 391 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.048947+0000 mon.a (mon.0) 1186 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.053593+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.057592+0000 mon.a (mon.0) 1188 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.064747+0000 mon.a (mon.0) 1189 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.072136+0000 mon.a (mon.0) 1190 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.075900+0000 mon.a (mon.0) 1191 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.080288+0000 mon.a (mon.0) 1192 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.192 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:33.332890+0000 mon.a (mon.0) 1193 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:52:34.690 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:34] "GET /metrics HTTP/1.1" 200 164571 "" "Prometheus/2.33.4" 2024-02-10T09:52:35.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:34.017708+0000 mgr.y (mgr.25138) 392 : cluster [DBG] pgmap v237: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 255 B/s wr, 162 op/s 2024-02-10T09:52:35.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:34.017708+0000 mgr.y (mgr.25138) 392 : cluster [DBG] pgmap v237: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 255 B/s wr, 162 op/s 2024-02-10T09:52:37.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:36.018246+0000 mgr.y (mgr.25138) 393 : cluster [DBG] pgmap v238: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 255 B/s wr, 170 op/s 2024-02-10T09:52:37.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:36.018246+0000 mgr.y (mgr.25138) 393 : cluster [DBG] pgmap v238: 129 pgs: 129 active+clean; 7.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 255 B/s wr, 170 op/s 2024-02-10T09:52:39.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:38.018834+0000 mgr.y (mgr.25138) 394 : cluster [DBG] pgmap v239: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 255 B/s wr, 128 op/s 2024-02-10T09:52:39.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:38.018834+0000 mgr.y (mgr.25138) 394 : cluster [DBG] pgmap v239: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 255 B/s wr, 128 op/s 2024-02-10T09:52:41.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:40.019319+0000 mgr.y (mgr.25138) 395 : cluster [DBG] pgmap v240: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 255 B/s wr, 88 op/s 2024-02-10T09:52:41.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:40.019319+0000 mgr.y (mgr.25138) 395 : cluster [DBG] pgmap v240: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 255 B/s wr, 88 op/s 2024-02-10T09:52:43.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:42.019960+0000 mgr.y (mgr.25138) 396 : cluster [DBG] pgmap v241: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 170 B/s wr, 79 op/s 2024-02-10T09:52:43.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:42.019960+0000 mgr.y (mgr.25138) 396 : cluster [DBG] pgmap v241: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 170 B/s wr, 79 op/s 2024-02-10T09:52:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:52:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:52:44.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:44] "GET /metrics HTTP/1.1" 200 169680 "" "Prometheus/2.33.4" 2024-02-10T09:52:45.692 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:44.020584+0000 mgr.y (mgr.25138) 397 : cluster [DBG] pgmap v242: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 170 B/s wr, 31 op/s 2024-02-10T09:52:45.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:44.020584+0000 mgr.y (mgr.25138) 397 : cluster [DBG] pgmap v242: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 170 B/s wr, 31 op/s 2024-02-10T09:52:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:46.021189+0000 mgr.y (mgr.25138) 398 : cluster [DBG] pgmap v243: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-02-10T09:52:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:46.021189+0000 mgr.y (mgr.25138) 398 : cluster [DBG] pgmap v243: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-02-10T09:52:49.484 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:48.021836+0000 mgr.y (mgr.25138) 399 : cluster [DBG] pgmap v244: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T09:52:49.484 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:48.021836+0000 mgr.y (mgr.25138) 399 : cluster [DBG] pgmap v244: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T09:52:50.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:52:50.588 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:49.454414+0000 mgr.y (mgr.25138) 400 : audit [DBG] from='client.54678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:50.589 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:49.454414+0000 mgr.y (mgr.25138) 400 : audit [DBG] from='client.54678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:51.415 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:50.022202+0000 mgr.y (mgr.25138) 401 : cluster [DBG] pgmap v245: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:51.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:50.022202+0000 mgr.y (mgr.25138) 401 : cluster [DBG] pgmap v245: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:52.096 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:52:52.096 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (20m) 20s ago 21m 26.7M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:52:52.096 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (8m) 19s ago 23m 57.9M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (9m) 19s ago 34m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (16m) 20s ago 36m 440M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (6m) 20s ago 36m 139M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (7m) 19s ago 34m 105M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (6m) 20s ago 34m 84.4M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (8m) 20s ago 26m 22.0M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (8m) 19s ago 26m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 20s ago 33m 47.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 20s ago 33m 47.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 20s ago 33m 44.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (3m) 20s ago 32m 45.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (2m) 19s ago 32m 44.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (2m) 19s ago 32m 45.3M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (115s) 19s ago 32m 43.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:52:52.097 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (86s) 19s ago 31m 44.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:52:52.098 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (21m) 19s ago 21m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:52:52.098 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (28s) 20s ago 23m 42.6M - 16.2.14-521-g0e714d9a 1fbc92c859dc 59737827741f 2024-02-10T09:52:52.098 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (23s) 19s ago 23m 39.9M - 16.2.14-521-g0e714d9a 1fbc92c859dc 995d6e382291 2024-02-10T09:52:52.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.085530+0000 mon.a (mon.0) 1194 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:52.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.085926+0000 mon.a (mon.0) 1195 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.086242+0000 mon.a (mon.0) 1196 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.086578+0000 mon.a (mon.0) 1197 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.086910+0000 mon.a (mon.0) 1198 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.087238+0000 mon.a (mon.0) 1199 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.088946+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.089285+0000 mon.a (mon.0) 1201 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.089545+0000 mon.a (mon.0) 1202 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.089798+0000 mon.a (mon.0) 1203 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.090049+0000 mon.a (mon.0) 1204 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.432 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.085530+0000 mon.a (mon.0) 1194 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.085926+0000 mon.a (mon.0) 1195 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.086242+0000 mon.a (mon.0) 1196 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.086578+0000 mon.a (mon.0) 1197 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.086910+0000 mon.a (mon.0) 1198 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.087238+0000 mon.a (mon.0) 1199 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.088946+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.089285+0000 mon.a (mon.0) 1201 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.089545+0000 mon.a (mon.0) 1202 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.089798+0000 mon.a (mon.0) 1203 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.433 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.090049+0000 mon.a (mon.0) 1204 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:52:52.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-02-10T09:52:53.370 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:52.022698+0000 mgr.y (mgr.25138) 402 : cluster [DBG] pgmap v246: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:53.370 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:52.083641+0000 mgr.y (mgr.25138) 403 : audit [DBG] from='client.64633 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:53.370 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:52.022698+0000 mgr.y (mgr.25138) 402 : cluster [DBG] pgmap v246: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:53.370 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:52.083641+0000 mgr.y (mgr.25138) 403 : audit [DBG] from='client.64633 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:52:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:52:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:52:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:52:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:52:54] "GET /metrics HTTP/1.1" 200 169683 "" "Prometheus/2.33.4" 2024-02-10T09:52:54.721 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:52:55.359 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:54.023232+0000 mgr.y (mgr.25138) 404 : cluster [DBG] pgmap v247: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:55.359 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:54.709155+0000 mon.c (mon.1) 17 : audit [DBG] from='client.64639 172.21.15.153:0/3481380255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:55.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:54.023232+0000 mgr.y (mgr.25138) 404 : cluster [DBG] pgmap v247: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:55.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:54.709155+0000 mon.c (mon.1) 17 : audit [DBG] from='client.64639 172.21.15.153:0/3481380255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:55.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-02-10T09:52:57.393 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)" 2024-02-10T09:52:57.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:56.023763+0000 mgr.y (mgr.25138) 405 : cluster [DBG] pgmap v248: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:57.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:56.023763+0000 mgr.y (mgr.25138) 405 : cluster [DBG] pgmap v248: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:58.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T09:52:58.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:52:57.380658+0000 mon.a (mon.0) 1205 : audit [DBG] from='client.64645 172.21.15.153:0/1498692083' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:58.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:52:57.380658+0000 mon.a (mon.0) 1205 : audit [DBG] from='client.64645 172.21.15.153:0/1498692083' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:52:59.412 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:52:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:52:58.024310+0000 mgr.y (mgr.25138) 406 : cluster [DBG] pgmap v249: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:52:59.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:52:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:52:58.024310+0000 mgr.y (mgr.25138) 406 : cluster [DBG] pgmap v249: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:00.064 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:53:00.607 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T09:53:00.616 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi153.front.sepia.ceph.com 2024-02-10T09:53:00.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:00.024818+0000 mgr.y (mgr.25138) 407 : cluster [DBG] pgmap v250: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:00.059229+0000 mgr.y (mgr.25138) 408 : audit [DBG] from='client.54702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:00.059790+0000 mgr.y (mgr.25138) 409 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:00.063377+0000 mon.a (mon.0) 1206 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:00.066971+0000 mon.a (mon.0) 1207 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:00.356794+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:00.359757+0000 mgr.y (mgr.25138) 410 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:00.024818+0000 mgr.y (mgr.25138) 407 : cluster [DBG] pgmap v250: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:01.326 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:00.059229+0000 mgr.y (mgr.25138) 408 : audit [DBG] from='client.54702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:01.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:00.059790+0000 mgr.y (mgr.25138) 409 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:53:01.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:00.063377+0000 mon.a (mon.0) 1206 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:01.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:00.066971+0000 mon.a (mon.0) 1207 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:53:01.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:00.356794+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:01.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:00.359757+0000 mgr.y (mgr.25138) 410 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:53:02.661 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:53:03.257 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:02.025326+0000 mgr.y (mgr.25138) 411 : cluster [DBG] pgmap v251: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:03.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:02.025326+0000 mgr.y (mgr.25138) 411 : cluster [DBG] pgmap v251: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (21m) 31s ago 21m 26.7M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (8m) 30s ago 24m 57.9M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (9m) 30s ago 34m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (17m) 31s ago 36m 440M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (7m) 31s ago 36m 139M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (7m) 30s ago 34m 105M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (6m) 31s ago 35m 84.4M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (9m) 31s ago 26m 22.0M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (9m) 30s ago 26m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 31s ago 34m 47.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 31s ago 33m 47.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:53:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 31s ago 33m 44.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (3m) 31s ago 33m 45.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (3m) 30s ago 32m 44.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (2m) 30s ago 32m 45.3M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (2m) 30s ago 32m 43.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (97s) 30s ago 31m 44.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (21m) 30s ago 21m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (39s) 31s ago 24m 42.6M - 16.2.14-521-g0e714d9a 1fbc92c859dc 59737827741f 2024-02-10T09:53:03.308 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (34s) 30s ago 23m 39.9M - 16.2.14-521-g0e714d9a 1fbc92c859dc 995d6e382291 2024-02-10T09:53:03.711 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:53:03.711 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:53:03.711 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:53:03.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 8 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 15 2024-02-10T09:53:03.712 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:53:03.713 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:53:04.028 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:53:04.028 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": null, 2024-02-10T09:53:04.028 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": false, 2024-02-10T09:53:04.028 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "", 2024-02-10T09:53:04.029 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-02-10T09:53:04.029 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": null, 2024-02-10T09:53:04.029 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "", 2024-02-10T09:53:04.029 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-02-10T09:53:04.029 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:53:04.082 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:53:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:53:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:02.649603+0000 mgr.y (mgr.25138) 412 : audit [DBG] from='client.54708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:02.977085+0000 mgr.y (mgr.25138) 413 : audit [DBG] from='client.54714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.295453+0000 mgr.y (mgr.25138) 414 : audit [DBG] from='client.54720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.297249+0000 mon.a (mon.0) 1209 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:04.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.297627+0000 mon.a (mon.0) 1210 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.297916+0000 mon.a (mon.0) 1211 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.298185+0000 mon.a (mon.0) 1212 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.298434+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.298697+0000 mon.a (mon.0) 1214 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.300324+0000 mon.a (mon.0) 1215 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.300590+0000 mon.a (mon.0) 1216 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.300849+0000 mon.a (mon.0) 1217 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.301097+0000 mon.a (mon.0) 1218 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.301345+0000 mon.a (mon.0) 1219 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.413 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:02.649603+0000 mgr.y (mgr.25138) 412 : audit [DBG] from='client.54708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:02.977085+0000 mgr.y (mgr.25138) 413 : audit [DBG] from='client.54714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.295453+0000 mgr.y (mgr.25138) 414 : audit [DBG] from='client.54720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.297249+0000 mon.a (mon.0) 1209 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.297627+0000 mon.a (mon.0) 1210 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.297916+0000 mon.a (mon.0) 1211 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.298185+0000 mon.a (mon.0) 1212 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.298434+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.298697+0000 mon.a (mon.0) 1214 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.300324+0000 mon.a (mon.0) 1215 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.300590+0000 mon.a (mon.0) 1216 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.300849+0000 mon.a (mon.0) 1217 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.301097+0000 mon.a (mon.0) 1218 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.414 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.301345+0000 mon.a (mon.0) 1219 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.588598+0000 mon.a (mon.0) 1220 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.588932+0000 mgr.y (mgr.25138) 415 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.588993+0000 mgr.y (mgr.25138) 416 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.589858+0000 mon.a (mon.0) 1221 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.597257+0000 mon.a (mon.0) 1222 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.598498+0000 mon.a (mon.0) 1223 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.599000+0000 mgr.y (mgr.25138) 417 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.605621+0000 mon.a (mon.0) 1224 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.606814+0000 mon.a (mon.0) 1225 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.607324+0000 mgr.y (mgr.25138) 418 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:53:04.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.613873+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.614787+0000 mon.a (mon.0) 1227 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.615331+0000 mgr.y (mgr.25138) 419 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.621023+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.622593+0000 mon.a (mon.0) 1229 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.623129+0000 mgr.y (mgr.25138) 420 : cephadm [INF] Upgrade: Setting container_image for all osd 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.629370+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.588598+0000 mon.a (mon.0) 1220 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.588932+0000 mgr.y (mgr.25138) 415 : cephadm [INF] Upgrade: Target is version 16.2.14-521-g0e714d9a (pacific) 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.588993+0000 mgr.y (mgr.25138) 416 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc'] 2024-02-10T09:53:04.416 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.589858+0000 mon.a (mon.0) 1221 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.597257+0000 mon.a (mon.0) 1222 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.598498+0000 mon.a (mon.0) 1223 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.599000+0000 mgr.y (mgr.25138) 417 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.605621+0000 mon.a (mon.0) 1224 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.606814+0000 mon.a (mon.0) 1225 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.607324+0000 mgr.y (mgr.25138) 418 : cephadm [INF] Upgrade: Setting container_image for all mon 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.613873+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.614787+0000 mon.a (mon.0) 1227 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.615331+0000 mgr.y (mgr.25138) 419 : cephadm [INF] Upgrade: Setting container_image for all crash 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.621023+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.622593+0000 mon.a (mon.0) 1229 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.623129+0000 mgr.y (mgr.25138) 420 : cephadm [INF] Upgrade: Setting container_image for all osd 2024-02-10T09:53:04.417 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.629370+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.630896+0000 mon.a (mon.0) 1231 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.637152+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.638293+0000 mon.a (mon.0) 1233 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.645768+0000 mon.a (mon.0) 1234 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.646696+0000 mon.a (mon.0) 1235 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.653864+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.654756+0000 mon.a (mon.0) 1237 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.656182+0000 mon.a (mon.0) 1238 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.663505+0000 mon.a (mon.0) 1239 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.664394+0000 mon.a (mon.0) 1240 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.671598+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.418 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.672762+0000 mon.a (mon.0) 1242 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.674391+0000 mon.a (mon.0) 1243 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.675941+0000 mon.a (mon.0) 1244 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.677445+0000 mon.a (mon.0) 1245 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.679108+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.685837+0000 mon.a (mon.0) 1247 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.630896+0000 mon.a (mon.0) 1231 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.637152+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.638293+0000 mon.a (mon.0) 1233 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.419 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.645768+0000 mon.a (mon.0) 1234 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.646696+0000 mon.a (mon.0) 1235 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.653864+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.654756+0000 mon.a (mon.0) 1237 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.656182+0000 mon.a (mon.0) 1238 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.663505+0000 mon.a (mon.0) 1239 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.664394+0000 mon.a (mon.0) 1240 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.671598+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.672762+0000 mon.a (mon.0) 1242 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.674391+0000 mon.a (mon.0) 1243 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.675941+0000 mon.a (mon.0) 1244 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.677445+0000 mon.a (mon.0) 1245 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.679108+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T09:53:04.420 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.685837+0000 mon.a (mon.0) 1247 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.686445+0000 mon.a (mon.0) 1248 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.689793+0000 mon.a (mon.0) 1249 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.690392+0000 mon.a (mon.0) 1250 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.694621+0000 mon.a (mon.0) 1251 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.695215+0000 mon.a (mon.0) 1252 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.700062+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.700631+0000 mon.a (mon.0) 1254 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.705449+0000 mon.a (mon.0) 1255 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.706096+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.710846+0000 mon.a (mon.0) 1257 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.710953+0000 mon.a (mon.0) 1258 : audit [DBG] from='client.54726 172.21.15.153:0/2187862577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.711785+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:53:04.421 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.716920+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.717651+0000 mon.a (mon.0) 1261 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.686445+0000 mon.a (mon.0) 1248 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.689793+0000 mon.a (mon.0) 1249 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.690392+0000 mon.a (mon.0) 1250 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.694621+0000 mon.a (mon.0) 1251 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.695215+0000 mon.a (mon.0) 1252 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.700062+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.700631+0000 mon.a (mon.0) 1254 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.705449+0000 mon.a (mon.0) 1255 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.706096+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.710846+0000 mon.a (mon.0) 1257 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T09:53:04.422 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.710953+0000 mon.a (mon.0) 1258 : audit [DBG] from='client.54726 172.21.15.153:0/2187862577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.711785+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.716920+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.717651+0000 mon.a (mon.0) 1261 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.718107+0000 mon.a (mon.0) 1262 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.722098+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.722862+0000 mon.a (mon.0) 1264 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.727447+0000 mon.a (mon.0) 1265 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.728169+0000 mon.a (mon.0) 1266 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.728647+0000 mon.a (mon.0) 1267 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.729115+0000 mon.a (mon.0) 1268 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.423 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.729567+0000 mon.a (mon.0) 1269 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.730023+0000 mon.a (mon.0) 1270 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.733107+0000 mon.a (mon.0) 1271 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.734023+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.736280+0000 mon.a (mon.0) 1273 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.736719+0000 mon.a (mon.0) 1274 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:03.741820+0000 mon.a (mon.0) 1275 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.718107+0000 mon.a (mon.0) 1262 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.722098+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.722862+0000 mon.a (mon.0) 1264 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.727447+0000 mon.a (mon.0) 1265 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.728169+0000 mon.a (mon.0) 1266 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.728647+0000 mon.a (mon.0) 1267 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.729115+0000 mon.a (mon.0) 1268 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.729567+0000 mon.a (mon.0) 1269 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.730023+0000 mon.a (mon.0) 1270 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.733107+0000 mon.a (mon.0) 1271 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.734023+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.736280+0000 mon.a (mon.0) 1273 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.736719+0000 mon.a (mon.0) 1274 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:53:04.425 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:03.741820+0000 mon.a (mon.0) 1275 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:53:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:04] "GET /metrics HTTP/1.1" 200 169683 "" "Prometheus/2.33.4" 2024-02-10T09:53:05.439 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.631443+0000 mgr.y (mgr.25138) 421 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.638822+0000 mgr.y (mgr.25138) 422 : cephadm [INF] Upgrade: Setting container_image for all rgw 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.647211+0000 mgr.y (mgr.25138) 423 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.655279+0000 mgr.y (mgr.25138) 424 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.656668+0000 mgr.y (mgr.25138) 425 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.664908+0000 mgr.y (mgr.25138) 426 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.673293+0000 mgr.y (mgr.25138) 427 : cephadm [INF] Upgrade: Setting container_image for all node-exporter 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.674881+0000 mgr.y (mgr.25138) 428 : cephadm [INF] Upgrade: Setting container_image for all prometheus 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.676404+0000 mgr.y (mgr.25138) 429 : cephadm [INF] Upgrade: Setting container_image for all alertmanager 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.677928+0000 mgr.y (mgr.25138) 430 : cephadm [INF] Upgrade: Setting container_image for all grafana 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.678490+0000 mgr.y (mgr.25138) 431 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cephadm 2024-02-10T09:53:03.733653+0000 mgr.y (mgr.25138) 432 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:04.025856+0000 mgr.y (mgr.25138) 433 : cluster [DBG] pgmap v252: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:05.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:04.027883+0000 mgr.y (mgr.25138) 434 : audit [DBG] from='client.64681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.631443+0000 mgr.y (mgr.25138) 421 : cephadm [INF] Upgrade: Setting container_image for all mds 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.638822+0000 mgr.y (mgr.25138) 422 : cephadm [INF] Upgrade: Setting container_image for all rgw 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.647211+0000 mgr.y (mgr.25138) 423 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.655279+0000 mgr.y (mgr.25138) 424 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.656668+0000 mgr.y (mgr.25138) 425 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.664908+0000 mgr.y (mgr.25138) 426 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.673293+0000 mgr.y (mgr.25138) 427 : cephadm [INF] Upgrade: Setting container_image for all node-exporter 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.674881+0000 mgr.y (mgr.25138) 428 : cephadm [INF] Upgrade: Setting container_image for all prometheus 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.676404+0000 mgr.y (mgr.25138) 429 : cephadm [INF] Upgrade: Setting container_image for all alertmanager 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.677928+0000 mgr.y (mgr.25138) 430 : cephadm [INF] Upgrade: Setting container_image for all grafana 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.678490+0000 mgr.y (mgr.25138) 431 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-02-10T09:53:05.441 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cephadm 2024-02-10T09:53:03.733653+0000 mgr.y (mgr.25138) 432 : cephadm [INF] Upgrade: Complete! 2024-02-10T09:53:05.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:04.025856+0000 mgr.y (mgr.25138) 433 : cluster [DBG] pgmap v252: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:05.442 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:04.027883+0000 mgr.y (mgr.25138) 434 : audit [DBG] from='client.64681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:07.439 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:06.026534+0000 mgr.y (mgr.25138) 435 : cluster [DBG] pgmap v253: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:07.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:06.026534+0000 mgr.y (mgr.25138) 435 : cluster [DBG] pgmap v253: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:08.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:07.941474+0000 mon.a (mon.0) 1276 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-10T09:53:08.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:07.941806+0000 mon.a (mon.0) 1277 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:53:08.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:07.942038+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-02-10T09:53:08.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:07.942264+0000 mon.a (mon.0) 1279 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:53:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:07.941474+0000 mon.a (mon.0) 1276 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-10T09:53:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:07.941806+0000 mon.a (mon.0) 1277 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T09:53:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:07.942038+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-02-10T09:53:08.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:07.942264+0000 mon.a (mon.0) 1279 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T09:53:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:08.027307+0000 mgr.y (mgr.25138) 436 : cluster [DBG] pgmap v254: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:09.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:08.350034+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:08.027307+0000 mgr.y (mgr.25138) 436 : cluster [DBG] pgmap v254: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:08.350034+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:53:11.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:10.028026+0000 mgr.y (mgr.25138) 437 : cluster [DBG] pgmap v255: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:11.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:10.028026+0000 mgr.y (mgr.25138) 437 : cluster [DBG] pgmap v255: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:13.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:12.028649+0000 mgr.y (mgr.25138) 438 : cluster [DBG] pgmap v256: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:12.028649+0000 mgr.y (mgr.25138) 438 : cluster [DBG] pgmap v256: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:53:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:53:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:53:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:14] "GET /metrics HTTP/1.1" 200 169693 "" "Prometheus/2.33.4" 2024-02-10T09:53:15.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:14.029130+0000 mgr.y (mgr.25138) 439 : cluster [DBG] pgmap v257: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:15.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:14.029130+0000 mgr.y (mgr.25138) 439 : cluster [DBG] pgmap v257: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:17.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:16.029674+0000 mgr.y (mgr.25138) 440 : cluster [DBG] pgmap v258: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:17.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:16.029674+0000 mgr.y (mgr.25138) 440 : cluster [DBG] pgmap v258: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:19.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:18.030405+0000 mgr.y (mgr.25138) 441 : cluster [DBG] pgmap v259: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:19.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:18.030405+0000 mgr.y (mgr.25138) 441 : cluster [DBG] pgmap v259: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:21.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:20.031098+0000 mgr.y (mgr.25138) 442 : cluster [DBG] pgmap v260: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:21.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:20.031098+0000 mgr.y (mgr.25138) 442 : cluster [DBG] pgmap v260: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:23.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:22.031827+0000 mgr.y (mgr.25138) 443 : cluster [DBG] pgmap v261: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:23.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:22.031827+0000 mgr.y (mgr.25138) 443 : cluster [DBG] pgmap v261: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:53:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:53:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:53:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:24] "GET /metrics HTTP/1.1" 200 169706 "" "Prometheus/2.33.4" 2024-02-10T09:53:25.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:24.032342+0000 mgr.y (mgr.25138) 444 : cluster [DBG] pgmap v262: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:25.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:24.032342+0000 mgr.y (mgr.25138) 444 : cluster [DBG] pgmap v262: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:27.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:26.032902+0000 mgr.y (mgr.25138) 445 : cluster [DBG] pgmap v263: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:27.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:26.032902+0000 mgr.y (mgr.25138) 445 : cluster [DBG] pgmap v263: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:29.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:28.033595+0000 mgr.y (mgr.25138) 446 : cluster [DBG] pgmap v264: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:29.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:28.033595+0000 mgr.y (mgr.25138) 446 : cluster [DBG] pgmap v264: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:31.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:30.034142+0000 mgr.y (mgr.25138) 447 : cluster [DBG] pgmap v265: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:31.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:30.034142+0000 mgr.y (mgr.25138) 447 : cluster [DBG] pgmap v265: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:33.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:32.034690+0000 mgr.y (mgr.25138) 448 : cluster [DBG] pgmap v266: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:33.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:32.034690+0000 mgr.y (mgr.25138) 448 : cluster [DBG] pgmap v266: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:53:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:53:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:53:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:34] "GET /metrics HTTP/1.1" 200 169706 "" "Prometheus/2.33.4" 2024-02-10T09:53:35.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:53:35.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:34.035385+0000 mgr.y (mgr.25138) 449 : cluster [DBG] pgmap v267: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:35.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:34.345423+0000 mgr.y (mgr.25138) 450 : audit [DBG] from='client.64687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:34.035385+0000 mgr.y (mgr.25138) 449 : cluster [DBG] pgmap v267: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:35.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:34.345423+0000 mgr.y (mgr.25138) 450 : audit [DBG] from='client.64687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (21m) 65s ago 22m 26.7M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (9m) 64s ago 24m 57.9M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (9m) 64s ago 35m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (17m) 65s ago 36m 440M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (7m) 65s ago 36m 139M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (8m) 64s ago 35m 105M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (7m) 65s ago 35m 84.4M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:53:37.104 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (9m) 65s ago 27m 22.0M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (9m) 64s ago 27m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (6m) 65s ago 34m 47.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 65s ago 34m 47.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 65s ago 34m 44.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (4m) 65s ago 33m 45.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (3m) 64s ago 33m 44.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (3m) 64s ago 33m 45.3M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (2m) 64s ago 32m 43.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (2m) 64s ago 32m 44.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (21m) 64s ago 21m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (73s) 65s ago 24m 42.6M - 16.2.14-521-g0e714d9a 1fbc92c859dc 59737827741f 2024-02-10T09:53:37.105 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (68s) 64s ago 24m 39.9M - 16.2.14-521-g0e714d9a 1fbc92c859dc 995d6e382291 2024-02-10T09:53:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:36.035831+0000 mgr.y (mgr.25138) 451 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.092018+0000 mon.a (mon.0) 1281 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.092339+0000 mon.a (mon.0) 1282 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.092612+0000 mon.a (mon.0) 1283 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.092910+0000 mon.a (mon.0) 1284 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.093156+0000 mon.a (mon.0) 1285 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.093392+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.094937+0000 mon.a (mon.0) 1287 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.095184+0000 mon.a (mon.0) 1288 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.095409+0000 mon.a (mon.0) 1289 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.095628+0000 mon.a (mon.0) 1290 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.095855+0000 mon.a (mon.0) 1291 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:36.035831+0000 mgr.y (mgr.25138) 451 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.092018+0000 mon.a (mon.0) 1281 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.092339+0000 mon.a (mon.0) 1282 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.092612+0000 mon.a (mon.0) 1283 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.092910+0000 mon.a (mon.0) 1284 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.093156+0000 mon.a (mon.0) 1285 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.093392+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.094937+0000 mon.a (mon.0) 1287 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.095184+0000 mon.a (mon.0) 1288 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.095409+0000 mon.a (mon.0) 1289 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.095628+0000 mon.a (mon.0) 1290 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.691 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.095855+0000 mon.a (mon.0) 1291 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:53:37.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:53:38.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:37.090112+0000 mgr.y (mgr.25138) 452 : audit [DBG] from='client.64693 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:38.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:37.090112+0000 mgr.y (mgr.25138) 452 : audit [DBG] from='client.64693 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:53:39.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:38.036292+0000 mgr.y (mgr.25138) 453 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:39.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:38.036292+0000 mgr.y (mgr.25138) 453 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:53:39.816 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 8 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 15 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:53:39.817 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:53:40.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-10T09:53:40.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:53:39.815099+0000 mon.a (mon.0) 1292 : audit [DBG] from='client.64699 172.21.15.153:0/3028980981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:40.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:40 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:53:39.815099+0000 mon.a (mon.0) 1292 : audit [DBG] from='client.64699 172.21.15.153:0/3028980981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:53:41.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:40.036763+0000 mgr.y (mgr.25138) 454 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:41.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:40.036763+0000 mgr.y (mgr.25138) 454 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:42.346 INFO:teuthology.orchestra.run.smithi153.stdout:wait for servicemap items w/ changing names to refresh 2024-02-10T09:53:42.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'sleep 60' 2024-02-10T09:53:43.623 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:42.037435+0000 mgr.y (mgr.25138) 455 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:43.623 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:42.037435+0000 mgr.y (mgr.25138) 455 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:44.083 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:53:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:53:44.579 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:53:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:44] "GET /metrics HTTP/1.1" 200 169702 "" "Prometheus/2.33.4" 2024-02-10T09:53:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:44.037950+0000 mgr.y (mgr.25138) 456 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:45.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:44.037950+0000 mgr.y (mgr.25138) 456 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:47.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:46.038537+0000 mgr.y (mgr.25138) 457 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:47.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:46.038537+0000 mgr.y (mgr.25138) 457 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:49.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:48.039143+0000 mgr.y (mgr.25138) 458 : cluster [DBG] pgmap v274: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:49.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:48.039143+0000 mgr.y (mgr.25138) 458 : cluster [DBG] pgmap v274: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:51.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:50.039686+0000 mgr.y (mgr.25138) 459 : cluster [DBG] pgmap v275: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:51.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:50.039686+0000 mgr.y (mgr.25138) 459 : cluster [DBG] pgmap v275: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:53.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:52.040238+0000 mgr.y (mgr.25138) 460 : cluster [DBG] pgmap v276: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:53.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:52.040238+0000 mgr.y (mgr.25138) 460 : cluster [DBG] pgmap v276: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:53:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:53:54.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:53:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:53:54] "GET /metrics HTTP/1.1" 200 169707 "" "Prometheus/2.33.4" 2024-02-10T09:53:55.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:54.041096+0000 mgr.y (mgr.25138) 461 : cluster [DBG] pgmap v277: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:55.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:54.041096+0000 mgr.y (mgr.25138) 461 : cluster [DBG] pgmap v277: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:57.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:56.041876+0000 mgr.y (mgr.25138) 462 : cluster [DBG] pgmap v278: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:57.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:56.041876+0000 mgr.y (mgr.25138) 462 : cluster [DBG] pgmap v278: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:59.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:53:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:53:58.042401+0000 mgr.y (mgr.25138) 463 : cluster [DBG] pgmap v279: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:53:59.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:53:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:53:58.042401+0000 mgr.y (mgr.25138) 463 : cluster [DBG] pgmap v279: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:01.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:00.043071+0000 mgr.y (mgr.25138) 464 : cluster [DBG] pgmap v280: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:01.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:00.043071+0000 mgr.y (mgr.25138) 464 : cluster [DBG] pgmap v280: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:03.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:02.043658+0000 mgr.y (mgr.25138) 465 : cluster [DBG] pgmap v281: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:03.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:02.043658+0000 mgr.y (mgr.25138) 465 : cluster [DBG] pgmap v281: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:54:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:54:04.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:54:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:04] "GET /metrics HTTP/1.1" 200 169707 "" "Prometheus/2.33.4" 2024-02-10T09:54:04.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:03.745546+0000 mon.a (mon.0) 1293 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:54:04.690 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:04.046397+0000 mon.a (mon.0) 1294 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:54:04.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:03.745546+0000 mon.a (mon.0) 1293 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:54:04.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:04.046397+0000 mon.a (mon.0) 1294 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:54:05.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:04.044015+0000 mgr.y (mgr.25138) 466 : cluster [DBG] pgmap v282: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:05.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:04.044015+0000 mgr.y (mgr.25138) 466 : cluster [DBG] pgmap v282: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:07.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:06.044543+0000 mgr.y (mgr.25138) 467 : cluster [DBG] pgmap v283: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:07.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:06.044543+0000 mgr.y (mgr.25138) 467 : cluster [DBG] pgmap v283: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:08.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:07.945506+0000 mon.a (mon.0) 1295 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:54:08.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:07.945834+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T09:54:08.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:07.946050+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-10T09:54:08.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:07.946263+0000 mon.a (mon.0) 1298 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:54:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:07.945506+0000 mon.a (mon.0) 1295 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-10T09:54:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:07.945834+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T09:54:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:07.946050+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-10T09:54:08.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:07.946263+0000 mon.a (mon.0) 1298 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T09:54:09.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:08.045153+0000 mgr.y (mgr.25138) 468 : cluster [DBG] pgmap v284: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:09.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:08.045153+0000 mgr.y (mgr.25138) 468 : cluster [DBG] pgmap v284: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:11.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:10.045779+0000 mgr.y (mgr.25138) 469 : cluster [DBG] pgmap v285: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:11.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:11 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:10.045779+0000 mgr.y (mgr.25138) 469 : cluster [DBG] pgmap v285: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:13.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:12.046617+0000 mgr.y (mgr.25138) 470 : cluster [DBG] pgmap v286: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:13.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:13 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:12.046617+0000 mgr.y (mgr.25138) 470 : cluster [DBG] pgmap v286: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:14.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:54:13 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:54:14.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:54:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:14] "GET /metrics HTTP/1.1" 200 169709 "" "Prometheus/2.33.4" 2024-02-10T09:54:15.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:14.047211+0000 mgr.y (mgr.25138) 471 : cluster [DBG] pgmap v287: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:15.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:15 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:14.047211+0000 mgr.y (mgr.25138) 471 : cluster [DBG] pgmap v287: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:17.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:16.047865+0000 mgr.y (mgr.25138) 472 : cluster [DBG] pgmap v288: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:17.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:17 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:16.047865+0000 mgr.y (mgr.25138) 472 : cluster [DBG] pgmap v288: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:19.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:18.048464+0000 mgr.y (mgr.25138) 473 : cluster [DBG] pgmap v289: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:19.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:19 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:18.048464+0000 mgr.y (mgr.25138) 473 : cluster [DBG] pgmap v289: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:21.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:20.048957+0000 mgr.y (mgr.25138) 474 : cluster [DBG] pgmap v290: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:21.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:20.048957+0000 mgr.y (mgr.25138) 474 : cluster [DBG] pgmap v290: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:23.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:22.049858+0000 mgr.y (mgr.25138) 475 : cluster [DBG] pgmap v291: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:23.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:23 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:22.049858+0000 mgr.y (mgr.25138) 475 : cluster [DBG] pgmap v291: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:24.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:54:23 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:54:24.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:54:24 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:24] "GET /metrics HTTP/1.1" 200 169706 "" "Prometheus/2.33.4" 2024-02-10T09:54:25.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:24.050528+0000 mgr.y (mgr.25138) 476 : cluster [DBG] pgmap v292: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:25.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:25 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:24.050528+0000 mgr.y (mgr.25138) 476 : cluster [DBG] pgmap v292: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:27.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:26.051138+0000 mgr.y (mgr.25138) 477 : cluster [DBG] pgmap v293: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:27.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:27 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:26.051138+0000 mgr.y (mgr.25138) 477 : cluster [DBG] pgmap v293: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:29.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:28.051819+0000 mgr.y (mgr.25138) 478 : cluster [DBG] pgmap v294: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:29.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:28.051819+0000 mgr.y (mgr.25138) 478 : cluster [DBG] pgmap v294: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:31.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:30.052472+0000 mgr.y (mgr.25138) 479 : cluster [DBG] pgmap v295: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:31 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:30.052472+0000 mgr.y (mgr.25138) 479 : cluster [DBG] pgmap v295: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:33.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:32.053151+0000 mgr.y (mgr.25138) 480 : cluster [DBG] pgmap v296: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:33.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:32.053151+0000 mgr.y (mgr.25138) 480 : cluster [DBG] pgmap v296: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:34.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:54:33 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:54:34.689 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:54:34 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:34] "GET /metrics HTTP/1.1" 200 169706 "" "Prometheus/2.33.4" 2024-02-10T09:54:35.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:34.053706+0000 mgr.y (mgr.25138) 481 : cluster [DBG] pgmap v297: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:35.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:35 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:34.053706+0000 mgr.y (mgr.25138) 481 : cluster [DBG] pgmap v297: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:37.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:36.054259+0000 mgr.y (mgr.25138) 482 : cluster [DBG] pgmap v298: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:37.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:37 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:36.054259+0000 mgr.y (mgr.25138) 482 : cluster [DBG] pgmap v298: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:39.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:38.055067+0000 mgr.y (mgr.25138) 483 : cluster [DBG] pgmap v299: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:39.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:39 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:38.055067+0000 mgr.y (mgr.25138) 483 : cluster [DBG] pgmap v299: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:41.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:40.055522+0000 mgr.y (mgr.25138) 484 : cluster [DBG] pgmap v300: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:41.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:41 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:40.055522+0000 mgr.y (mgr.25138) 484 : cluster [DBG] pgmap v300: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:43.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:42.056278+0000 mgr.y (mgr.25138) 485 : cluster [DBG] pgmap v301: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:43.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:43 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:42.056278+0000 mgr.y (mgr.25138) 485 : cluster [DBG] pgmap v301: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:44.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:54:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:54:44.684 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:54:44 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:44] "GET /metrics HTTP/1.1" 200 169719 "" "Prometheus/2.33.4" 2024-02-10T09:54:45.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ps' 2024-02-10T09:54:45.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:44.056979+0000 mgr.y (mgr.25138) 486 : cluster [DBG] pgmap v302: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:45.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:45 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:44.056979+0000 mgr.y (mgr.25138) 486 : cluster [DBG] pgmap v302: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:47.307 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.a smithi153 *:9093,9094 running (22m) 2m ago 23m 26.7M - 0.23.0 ba2b418f427c a318f98f94b4 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.a smithi204 running (10m) 2m ago 25m 57.9M - 8.3.5 dad864ee21e9 67a3139af0c0 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.x smithi204 *:8443,9283 running (11m) 2m ago 36m 387M - 16.2.14-521-g0e714d9a 1fbc92c859dc c26e3d68d449 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.y smithi153 *:8443,9283 running (18m) 2m ago 37m 440M - 16.2.14-521-g0e714d9a 1fbc92c859dc a9598abcef48 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a smithi153 running (8m) 2m ago 38m 139M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc 029c82f28e3c 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b smithi204 running (9m) 2m ago 36m 105M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc a2615d19aa22 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c smithi153 running (8m) 2m ago 36m 84.4M 2048M 16.2.14-521-g0e714d9a 1fbc92c859dc e4f1164dd253 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.a smithi153 running (10m) 2m ago 28m 22.0M - 1.3.1 1dbe0e931976 086dc26b8c2c 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.b smithi204 running (10m) 2m ago 28m 23.0M - 1.3.1 1dbe0e931976 accc637b7e99 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (7m) 2m ago 35m 47.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 6c831d5948b5 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (6m) 2m ago 35m 47.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 24b6f232a23e 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (6m) 2m ago 35m 44.6M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc a3f67a37aa30 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi153 running (5m) 2m ago 34m 45.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1200374c128b 2024-02-10T09:54:47.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi204 running (4m) 2m ago 34m 44.9M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 3b018a5578b9 2024-02-10T09:54:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi204 running (4m) 2m ago 34m 45.3M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc d9e9aab0c753 2024-02-10T09:54:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 smithi204 running (3m) 2m ago 34m 43.4M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 7e67b938cc88 2024-02-10T09:54:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 smithi204 running (3m) 2m ago 33m 44.0M 4096M 16.2.14-521-g0e714d9a 1fbc92c859dc 1832086c6649 2024-02-10T09:54:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.a smithi204 *:9095 running (22m) 2m ago 22m 74.7M - 2.33.4 514e6a882f6e ac1f43abcfcc 2024-02-10T09:54:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi153.zyveqt smithi153 running (2m) 2m ago 25m 42.6M - 16.2.14-521-g0e714d9a 1fbc92c859dc 59737827741f 2024-02-10T09:54:47.309 INFO:teuthology.orchestra.run.smithi153.stdout:rgw.r.z.smithi204.kmhqqg smithi204 running (2m) 2m ago 25m 39.9M - 16.2.14-521-g0e714d9a 1fbc92c859dc 995d6e382291 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:46.057547+0000 mgr.y (mgr.25138) 487 : cluster [DBG] pgmap v303: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.297936+0000 mon.a (mon.0) 1299 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.298291+0000 mon.a (mon.0) 1300 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.298551+0000 mon.a (mon.0) 1301 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.298813+0000 mon.a (mon.0) 1302 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.299066+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.299325+0000 mon.a (mon.0) 1304 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.300857+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.301109+0000 mon.a (mon.0) 1306 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.301386+0000 mon.a (mon.0) 1307 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.301640+0000 mon.a (mon.0) 1308 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.301901+0000 mon.a (mon.0) 1309 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:46.057547+0000 mgr.y (mgr.25138) 487 : cluster [DBG] pgmap v303: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.297936+0000 mon.a (mon.0) 1299 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.a","key": "mon_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.298291+0000 mon.a (mon.0) 1300 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.c","key": "mon_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.298551+0000 mon.a (mon.0) 1301 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.0","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.298813+0000 mon.a (mon.0) 1302 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.1","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.299066+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.2","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.299325+0000 mon.a (mon.0) 1304 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.3","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.300857+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "mon.b","key": "mon_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.301109+0000 mon.a (mon.0) 1306 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.4","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.301386+0000 mon.a (mon.0) 1307 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.5","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.301640+0000 mon.a (mon.0) 1308 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.6","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.645 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:47 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.301901+0000 mon.a (mon.0) 1309 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config get","who": "osd.7","key": "osd_memory_target"}]: dispatch 2024-02-10T09:54:47.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions' 2024-02-10T09:54:48.566 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:47.296089+0000 mgr.y (mgr.25138) 488 : audit [DBG] from='client.64705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:54:48.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:47.296089+0000 mgr.y (mgr.25138) 488 : audit [DBG] from='client.64705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:54:49.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:48.058081+0000 mgr.y (mgr.25138) 489 : cluster [DBG] pgmap v304: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:49.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:49 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:48.058081+0000 mgr.y (mgr.25138) 489 : cluster [DBG] pgmap v304: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:49.950 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:54:49.950 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-02-10T09:54:49.950 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 3 2024-02-10T09:54:49.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:54:49.950 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-02-10T09:54:49.950 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 8 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": {}, 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw": { 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 2 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)": 15 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-02-10T09:54:49.951 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:54:50.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-10T09:54:50.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:49.949931+0000 mon.a (mon.0) 1310 : audit [DBG] from='client.54762 172.21.15.153:0/2310639821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:54:50.817 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:50 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:49.949931+0000 mon.a (mon.0) 1310 : audit [DBG] from='client.54762 172.21.15.153:0/2310639821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:54:51.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:50.058644+0000 mgr.y (mgr.25138) 490 : cluster [DBG] pgmap v305: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:51.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:51 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:50.058644+0000 mgr.y (mgr.25138) 490 : cluster [DBG] pgmap v305: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:52.453 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-02-10T09:54:52.812 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:52.442029+0000 mon.a (mon.0) 1311 : audit [DBG] from='client.54768 172.21.15.153:0/665422778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:54:52.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:52 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:52.442029+0000 mon.a (mon.0) 1311 : audit [DBG] from='client.54768 172.21.15.153:0/665422778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:54:53.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-10T09:54:53.939 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:52.059463+0000 mgr.y (mgr.25138) 491 : cluster [DBG] pgmap v306: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:53.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:53 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:52.059463+0000 mgr.y (mgr.25138) 491 : cluster [DBG] pgmap v306: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:54.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:54:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:54:54.439 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:54:54 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:54:54] "GET /metrics HTTP/1.1" 200 169715 "" "Prometheus/2.33.4" 2024-02-10T09:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)" 2024-02-10T09:54:55.905 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:54.059984+0000 mgr.y (mgr.25138) 492 : cluster [DBG] pgmap v307: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:55.905 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:55.235191+0000 mon.a (mon.0) 1312 : audit [DBG] from='client.54774 172.21.15.153:0/3309369548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:54:55.906 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:54.059984+0000 mgr.y (mgr.25138) 492 : cluster [DBG] pgmap v307: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:55.906 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:55 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:55.235191+0000 mon.a (mon.0) 1312 : audit [DBG] from='client.54774 172.21.15.153:0/3309369548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T09:54:55.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-10T09:54:57.902 INFO:teuthology.orchestra.run.smithi153.stdout:osd 8 2m ago - 2024-02-10T09:54:57.924 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:56.060442+0000 mgr.y (mgr.25138) 493 : cluster [DBG] pgmap v308: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:57.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:57 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:56.060442+0000 mgr.y (mgr.25138) 493 : cluster [DBG] pgmap v308: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:58.514 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T09:54:58.524 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi153.front.sepia.ceph.com 2024-02-10T09:54:58.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-02-10T09:54:59.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:54:57.886478+0000 mgr.y (mgr.25138) 494 : audit [DBG] from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:54:59.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:54:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:54:58.061025+0000 mgr.y (mgr.25138) 495 : cluster [DBG] pgmap v309: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:54:59.775 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:54:57.886478+0000 mgr.y (mgr.25138) 494 : audit [DBG] from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:54:59.775 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:54:59 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:54:58.061025+0000 mgr.y (mgr.25138) 495 : cluster [DBG] pgmap v309: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "image": "quay.io/ceph/ceph", 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "registry": "quay.io", 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "bare_image": "ceph/ceph", 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "versions": [ 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "18.2.1", 2024-02-10T09:55:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "18.2.0", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "18.1.3", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "18.1.2", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "18.1.1", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "18.1.0", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.7", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.6", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.5", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.4", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.3", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.2", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.1", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.2.0", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "17.1.0", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.14", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.13", 2024-02-10T09:55:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.12", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.11", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.10", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.9", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.8", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.7", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.6", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.5", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.4", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.3", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.2", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.1", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.0", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.17", 2024-02-10T09:55:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.16", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.15", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.14", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.13", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.12", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.11", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.10", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.9", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.8", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.7", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.6", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.5", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.4", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.3", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.2", 2024-02-10T09:55:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.1", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "15.2.0", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.22", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.21", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.20", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.19", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.18", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.17", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.16", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.15", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.14", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.13", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.12", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.11", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.10", 2024-02-10T09:55:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.9", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.8", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.7", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.6", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.5", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.4", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.3", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.2", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.1", 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "14.2.0" 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-02-10T09:55:01.050 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-02-10T09:55:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:55:00.061438+0000 mgr.y (mgr.25138) 496 : cluster [DBG] pgmap v310: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:01.689 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:55:00.473496+0000 mgr.y (mgr.25138) 497 : audit [DBG] from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:55:01.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:55:00.061438+0000 mgr.y (mgr.25138) 496 : cluster [DBG] pgmap v310: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:01.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:01 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:55:00.473496+0000 mgr.y (mgr.25138) 497 : audit [DBG] from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:55:01.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph | grep 16.2.0' 2024-02-10T09:55:03.940 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:55:02.062005+0000 mgr.y (mgr.25138) 498 : cluster [DBG] pgmap v311: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:03.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:03 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:55:02.062005+0000 mgr.y (mgr.25138) 498 : cluster [DBG] pgmap v311: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:04.081 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:55:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x[56362]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:55:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T09:55:04.226 INFO:teuthology.orchestra.run.smithi153.stdout: "16.2.0", 2024-02-10T09:55:04.580 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: ::ffff:172.21.15.204 - - [10/Feb/2024:09:55:04] "GET /metrics HTTP/1.1" 200 169715 "" "Prometheus/2.33.4" 2024-02-10T09:55:04.894 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:55:04.049946+0000 mon.a (mon.0) 1313 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:55:04.894 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:55:04.353750+0000 mon.a (mon.0) 1314 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:55:04.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:55:04.049946+0000 mon.a (mon.0) 1313 : audit [DBG] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:55:04.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:04 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:55:04.353750+0000 mon.a (mon.0) 1314 : audit [INF] from='mgr.25138 172.21.15.153:0/4028809393' entity='mgr.y' 2024-02-10T09:55:04.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-02-10T09:55:05.870 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: audit 2024-02-10T09:55:03.669604+0000 mgr.y (mgr.25138) 499 : audit [DBG] from='client.54792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:55:05.870 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:55:04.062449+0000 mgr.y (mgr.25138) 500 : cluster [DBG] pgmap v312: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:05.870 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: audit 2024-02-10T09:55:03.669604+0000 mgr.y (mgr.25138) 499 : audit [DBG] from='client.54792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:55:05.870 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:05 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:55:04.062449+0000 mgr.y (mgr.25138) 500 : cluster [DBG] pgmap v312: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:07.435 INFO:teuthology.orchestra.run.smithi153.stdout: "v16.2.2", 2024-02-10T09:55:07.435 INFO:teuthology.orchestra.run.smithi153.stdout: "v16.2.2-20210505", 2024-02-10T09:55:07.759 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: cluster 2024-02-10T09:55:06.062930+0000 mgr.y (mgr.25138) 501 : cluster [DBG] pgmap v313: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:07.759 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:07 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: cluster 2024-02-10T09:55:06.062930+0000 mgr.y (mgr.25138) 501 : cluster [DBG] pgmap v313: 129 pgs: 129 active+clean; 7.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T09:55:08.056 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T09:55:08.066 INFO:tasks.cephadm:Teardown begin 2024-02-10T09:55:08.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:55:08.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:55:08.141 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T09:55:08.141 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:55:08.162 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:55:08.202 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T09:55:08.202 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T09:55:08.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a 2024-02-10T09:55:08.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:08 smithi153 systemd[1]: Stopping Ceph mon.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:08.440 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:08 smithi153 bash[93773]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T09:55:08.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:55:08.556+0000 7ff668c65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T09:55:08.873 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a[72128]: debug 2024-02-10T09:55:08.556+0000 7ff668c65700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-02-10T09:55:08.874 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:08] ENGINE Bus STOPPING 2024-02-10T09:55:08.874 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:55:08.874 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:08] ENGINE Bus STOPPED 2024-02-10T09:55:08.874 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:08] ENGINE Bus STARTING 2024-02-10T09:55:08.874 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:08] ENGINE Serving on http://:::9283 2024-02-10T09:55:08.874 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:08] ENGINE Bus STARTED 2024-02-10T09:55:09.189 INFO:journalctl@ceph.mon.a.smithi153.stdout:Feb 10 09:55:08 smithi153 bash[93812]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-a 2024-02-10T09:55:09.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T09:55:09.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:09.415 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T09:55:09.415 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-10T09:55:09.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.c 2024-02-10T09:55:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:09 smithi153 systemd[1]: Stopping Ceph mon.c for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:09.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:09 smithi153 bash[93944]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.c" found: no such container 2024-02-10T09:55:10.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:55:09.704+0000 7feba2348700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T09:55:10.135 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c[73450]: debug 2024-02-10T09:55:09.705+0000 7feba2348700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-02-10T09:55:10.137 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:09] ENGINE Bus STOPPING 2024-02-10T09:55:10.137 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T09:55:10.137 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:09] ENGINE Bus STOPPED 2024-02-10T09:55:10.137 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:09] ENGINE Bus STARTING 2024-02-10T09:55:10.137 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:09] ENGINE Serving on http://:::9283 2024-02-10T09:55:10.137 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:09 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: [10/Feb/2024:09:55:09] ENGINE Bus STARTED 2024-02-10T09:55:10.440 INFO:journalctl@ceph.mon.c.smithi153.stdout:Feb 10 09:55:10 smithi153 bash[93982]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-c 2024-02-10T09:55:10.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T09:55:10.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:10.579 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-10T09:55:10.579 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T09:55:10.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.b 2024-02-10T09:55:10.915 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:55:10 smithi204 systemd[1]: Stopping Ceph mon.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:10.915 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:55:10 smithi204 bash[72208]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon.b" found: no such container 2024-02-10T09:55:11.249 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:55:10 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:55:10.913+0000 7fa01971e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T09:55:11.249 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:55:10 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b[60103]: debug 2024-02-10T09:55:10.913+0000 7fa01971e700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-02-10T09:55:11.581 INFO:journalctl@ceph.mon.b.smithi204.stdout:Feb 10 09:55:11 smithi204 bash[72249]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mon-b 2024-02-10T09:55:11.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T09:55:11.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:11.877 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T09:55:11.877 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T09:55:11.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T09:55:12.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:11 smithi153 systemd[1]: Stopping Ceph mgr.y for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:12.189 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:11 smithi153 bash[94112]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.y" found: no such container 2024-02-10T09:55:12.497 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:12 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:55:12.189+0000 7fc5251f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:12.497 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:12 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y[62030]: debug 2024-02-10T09:55:12.189+0000 7fc5251f6700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T09:55:12.939 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Feb 10 09:55:12 smithi153 bash[94149]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-y 2024-02-10T09:55:12.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T09:55:13.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:13.031 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T09:55:13.031 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T09:55:13.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x 2024-02-10T09:55:13.331 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:55:13 smithi204 systemd[1]: Stopping Ceph mgr.x for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:13.332 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:55:13 smithi204 bash[72369]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T09:55:13.913 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Feb 10 09:55:13 smithi204 bash[72409]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-mgr-x 2024-02-10T09:55:14.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T09:55:14.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:14.149 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T09:55:14.149 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T09:55:14.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.0 2024-02-10T09:55:14.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:55:14 smithi153 systemd[1]: Stopping Ceph osd.0 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:14.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:55:14 smithi153 bash[94268]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T09:55:14.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:55:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:55:14.425+0000 7fed75bd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:14.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:55:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:55:14.425+0000 7fed75bd4700 -1 osd.0 114 *** Got signal Terminated *** 2024-02-10T09:55:14.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:55:14 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0[77293]: debug 2024-02-10T09:55:14.425+0000 7fed75bd4700 -1 osd.0 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:55:18.940 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:18.675Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:18.940 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:18.676Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:55:20.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Feb 10 09:55:19 smithi153 bash[94306]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-0 2024-02-10T09:55:21.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T09:55:21.547 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:21.547 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T09:55:21.547 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T09:55:21.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.1 2024-02-10T09:55:21.899 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:55:21 smithi153 systemd[1]: Stopping Ceph osd.1 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:21.899 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:55:21 smithi153 bash[94541]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T09:55:22.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:55:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:55:21.896+0000 7f68380f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:22.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:55:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:55:21.896+0000 7f68380f1700 -1 osd.1 114 *** Got signal Terminated *** 2024-02-10T09:55:22.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:55:21 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1[79057]: debug 2024-02-10T09:55:21.896+0000 7f68380f1700 -1 osd.1 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:55:27.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Feb 10 09:55:27 smithi153 bash[94581]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-1 2024-02-10T09:55:28.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:55:28.675Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:28.940 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:28.676Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:28.940 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:28.676Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:55:29.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T09:55:29.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:29.062 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T09:55:29.062 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T09:55:29.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.2 2024-02-10T09:55:29.399 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:29 smithi153 systemd[1]: Stopping Ceph osd.2 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:29.399 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:29 smithi153 bash[94823]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T09:55:29.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:55:29.397+0000 7fa1a1e52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:29.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:55:29.397+0000 7fa1a1e52700 -1 osd.2 114 *** Got signal Terminated *** 2024-02-10T09:55:29.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:29 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2[81960]: debug 2024-02-10T09:55:29.397+0000 7fa1a1e52700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:55:34.939 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:34 smithi153 bash[94864]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-2 2024-02-10T09:55:36.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T09:55:36.589 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:36 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.2.service: Succeeded. 2024-02-10T09:55:36.589 INFO:journalctl@ceph.osd.2.smithi153.stdout:Feb 10 09:55:36 smithi153 systemd[1]: Stopped Ceph osd.2 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:55:36.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:36.601 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T09:55:36.601 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T09:55:36.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.3 2024-02-10T09:55:36.933 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:36 smithi153 systemd[1]: Stopping Ceph osd.3 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:36.933 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:36 smithi153 bash[95105]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T09:55:37.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:55:36.931+0000 7f2a50d16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:37.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:55:36.931+0000 7f2a50d16700 -1 osd.3 114 *** Got signal Terminated *** 2024-02-10T09:55:37.189 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:36 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3[84825]: debug 2024-02-10T09:55:36.931+0000 7f2a50d16700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:55:38.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:55:38.676Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:38.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:38.676Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:38.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:38 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:38.677Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:55:40.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:39 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:39.925+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:41.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:40 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:40.877+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:42.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:41 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:41.922+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:42.439 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:42 smithi153 bash[95144]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-3 2024-02-10T09:55:43.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:42 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:42.969+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:43.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T09:55:43.939 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:43 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.3.service: Succeeded. 2024-02-10T09:55:43.939 INFO:journalctl@ceph.osd.3.smithi153.stdout:Feb 10 09:55:43 smithi153 systemd[1]: Stopped Ceph osd.3 for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:55:43.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:43.954 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T09:55:43.955 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T09:55:43.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.4 2024-02-10T09:55:44.290 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:43 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:43.967+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:44.290 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:43 smithi204 systemd[1]: Stopping Ceph osd.4 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:44.290 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:44 smithi204 bash[72533]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T09:55:44.581 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:44.289+0000 7fe25d299700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:44.581 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:44.289+0000 7fe25d299700 -1 osd.4 114 *** Got signal Terminated *** 2024-02-10T09:55:44.582 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:44.289+0000 7fe25d299700 -1 osd.4 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:55:44.973 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:44.715+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.688472+0000 front 2024-02-10T09:55:18.688324+0000 (oldest deadline 2024-02-10T09:55:43.988325+0000) 2024-02-10T09:55:45.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:45.022+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:45.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:44 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:44.972+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:45.973 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:45.654+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:45.974 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:45.680+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.688472+0000 front 2024-02-10T09:55:18.688324+0000 (oldest deadline 2024-02-10T09:55:43.988325+0000) 2024-02-10T09:55:46.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:46.028+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:46.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:45 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:45.973+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:46.993 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:46.631+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.688472+0000 front 2024-02-10T09:55:18.688324+0000 (oldest deadline 2024-02-10T09:55:43.988325+0000) 2024-02-10T09:55:46.994 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:46.693+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:47.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:47.010+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:47.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:46 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:46.993+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:47.953 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:47.657+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.688472+0000 front 2024-02-10T09:55:18.688324+0000 (oldest deadline 2024-02-10T09:55:43.988325+0000) 2024-02-10T09:55:47.954 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:47.657+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.889346+0000 front 2024-02-10T09:55:26.889378+0000 (oldest deadline 2024-02-10T09:55:47.389127+0000) 2024-02-10T09:55:47.954 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:47.742+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:48.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:48.000+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:48.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:47.952+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:48.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:47 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:47.952+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:48.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:55:48.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:48.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:48.677Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:48.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:48 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:48.678Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:55:48.971 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:48.768+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:48.971 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:48.698+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.688472+0000 front 2024-02-10T09:55:18.688324+0000 (oldest deadline 2024-02-10T09:55:43.988325+0000) 2024-02-10T09:55:48.971 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4[63254]: debug 2024-02-10T09:55:48.698+0000 7fe25a216700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.889346+0000 front 2024-02-10T09:55:26.889378+0000 (oldest deadline 2024-02-10T09:55:47.389127+0000) 2024-02-10T09:55:49.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:49.032+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:49.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:49.032+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:49.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:48.969+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:49.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:48 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:48.969+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:49.978 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:49.728+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:49.978 INFO:journalctl@ceph.osd.4.smithi204.stdout:Feb 10 09:55:49 smithi204 bash[72574]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-4 2024-02-10T09:55:50.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:49.985+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:50.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:49.985+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:50.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:49.976+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:50.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:49 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:49.976+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:50.975 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:50 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:50.681+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:50.975 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:50 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:50.681+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:51.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:50 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:50.973+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:51.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:50 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:50.973+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:51.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:50 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:50.979+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:51.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:50 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:50.979+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:51.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T09:55:51.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:51.474 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T09:55:51.474 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T09:55:51.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.5 2024-02-10T09:55:51.825 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:51 smithi204 systemd[1]: Stopping Ceph osd.5 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:51.826 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:51 smithi204 bash[72815]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T09:55:51.826 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:51 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:51.669+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:51.826 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:51 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:51.669+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:52 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:52.029+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:52 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:52.029+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:51 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:51.824+0000 7f52facc2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:51 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:51.824+0000 7f52facc2700 -1 osd.5 114 *** Got signal Terminated *** 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:51 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:51.824+0000 7f52facc2700 -1 osd.5 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:52 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:52.010+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:52.082 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:52 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:52.010+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:53.035 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:52 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:52.700+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:53.035 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:52 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:52.700+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:53.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:53.033+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:53.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:53.033+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:53.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:53.074+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:53.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:53.074+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:53.987 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:53.703+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:53.987 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:53.703+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:54.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:54 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:54.088+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:54.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:54 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:54.088+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:54.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:53.985+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:54.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:53 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:53.985+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:55.010 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:54 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:54.700+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:55.010 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:54 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:54.700+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:55.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:55.114+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:55.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:55.114+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:55.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:55.008+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:55.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:55.008+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:55.332 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:55.008+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:29.421289+0000 front 2024-02-10T09:55:29.421229+0000 (oldest deadline 2024-02-10T09:55:54.721278+0000) 2024-02-10T09:55:56.044 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:55.710+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:56.044 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:55 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:55.710+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:56.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:56.043+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:18.320159+0000 front 2024-02-10T09:55:18.320267+0000 (oldest deadline 2024-02-10T09:55:44.220076+0000) 2024-02-10T09:55:56.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:56.043+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.220619+0000 front 2024-02-10T09:55:24.220459+0000 (oldest deadline 2024-02-10T09:55:48.920431+0000) 2024-02-10T09:55:56.331 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5[64940]: debug 2024-02-10T09:55:56.043+0000 7f52f8440700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:29.421289+0000 front 2024-02-10T09:55:29.421229+0000 (oldest deadline 2024-02-10T09:55:54.721278+0000) 2024-02-10T09:55:56.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:56.116+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:56.332 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:56.116+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:57.076 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:56.759+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:57.076 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:56.759+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:57.076 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:56 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:56.759+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:55:57.330 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:57.073+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:57.330 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:57.074+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:57.581 INFO:journalctl@ceph.osd.5.smithi204.stdout:Feb 10 09:55:57 smithi204 bash[72855]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-5 2024-02-10T09:55:58.072 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:57.734+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:58.072 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:57.734+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:58.072 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:57 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:57.734+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:55:58.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:58.104+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:58.331 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:58.104+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:58.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:55:58.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:55:58.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:58.677Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:55:58.940 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:55:58 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:55:58.678Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:55:59.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:58.770+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:55:59.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:58.770+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:55:59.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:58 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:58.770+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:55:59.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T09:55:59.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:55:59.139 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T09:55:59.139 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T09:55:59.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.6 2024-02-10T09:55:59.478 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:59.094+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:55:59.478 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:59.095+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:55:59.478 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:59.095+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:33.556149+0000 front 2024-02-10T09:55:33.556245+0000 (oldest deadline 2024-02-10T09:55:58.256127+0000) 2024-02-10T09:55:59.478 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 systemd[1]: Stopping Ceph osd.6 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:55:59.478 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 bash[73093]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T09:55:59.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:59.475+0000 7f04abd53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:55:59.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:59.475+0000 7f04abd53700 -1 osd.6 114 *** Got signal Terminated *** 2024-02-10T09:55:59.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:55:59.475+0000 7f04abd53700 -1 osd.6 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:56:00.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:59.780+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:00.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:59.780+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:00.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:55:59 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:55:59.780+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:00.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:00.119+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:56:00.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:00.119+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:56:00.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:00.119+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:33.556149+0000 front 2024-02-10T09:55:33.556245+0000 (oldest deadline 2024-02-10T09:55:58.256127+0000) 2024-02-10T09:56:01.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:00.778+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:01.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:00.778+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:01.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:00 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:00.778+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:01.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:01.118+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:56:01.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:01.118+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:56:01.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:01.118+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:33.556149+0000 front 2024-02-10T09:55:33.556245+0000 (oldest deadline 2024-02-10T09:55:58.256127+0000) 2024-02-10T09:56:02.085 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:01.794+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:02.085 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:01.794+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:02.085 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:01 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:01.794+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:02.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:02.113+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:56:02.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:02.113+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:56:02.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:02.113+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:33.556149+0000 front 2024-02-10T09:55:33.556245+0000 (oldest deadline 2024-02-10T09:55:58.256127+0000) 2024-02-10T09:56:03.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:02.765+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:03.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:02.765+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:03.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:02 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:02.765+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:03.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:03.089+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:56:03.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:03.089+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:56:03.582 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:03.089+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:33.556149+0000 front 2024-02-10T09:55:33.556245+0000 (oldest deadline 2024-02-10T09:55:58.256127+0000) 2024-02-10T09:56:04.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:03.718+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:04.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:03.718+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:04.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:03 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:03.718+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:04.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:04 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:04.119+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:14.552119+0000 front 2024-02-10T09:55:14.552066+0000 (oldest deadline 2024-02-10T09:55:39.852209+0000) 2024-02-10T09:56:04.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:04 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:04.119+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:26.154032+0000 front 2024-02-10T09:55:26.154093+0000 (oldest deadline 2024-02-10T09:55:47.254047+0000) 2024-02-10T09:56:04.581 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:04 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6[66648]: debug 2024-02-10T09:56:04.119+0000 7f04a4cc8700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:33.556149+0000 front 2024-02-10T09:55:33.556245+0000 (oldest deadline 2024-02-10T09:55:58.256127+0000) 2024-02-10T09:56:05.081 INFO:journalctl@ceph.osd.6.smithi204.stdout:Feb 10 09:56:04 smithi204 bash[73134]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-6 2024-02-10T09:56:05.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:04 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:04.724+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:05.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:04 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:04.724+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:05.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:04 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:04.724+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:06.004 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:05 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:05.695+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:06.005 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:05 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:05.695+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:06.005 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:05 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:05.695+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:06.733 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:06 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:06.692+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:06.734 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:06 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:06.692+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:06.734 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:06 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:06.692+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:06.734 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:06 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:06.692+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6830 osd.3 since back 2024-02-10T09:55:41.055732+0000 front 2024-02-10T09:55:41.055720+0000 (oldest deadline 2024-02-10T09:56:05.755786+0000) 2024-02-10T09:56:06.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T09:56:06.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:56:06.786 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T09:56:06.786 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T09:56:06.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.7 2024-02-10T09:56:07.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:06 smithi204 systemd[1]: Stopping Ceph osd.7 for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:56:07.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:06 smithi204 bash[73381]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T09:56:07.581 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.127+0000 7f7ac737d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T09:56:07.581 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.127+0000 7f7ac737d700 -1 osd.7 114 *** Got signal Terminated *** 2024-02-10T09:56:07.581 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.127+0000 7f7ac737d700 -1 osd.7 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T09:56:08.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.741+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:08.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.741+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:08.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.741+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:08.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:07 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:07.741+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6830 osd.3 since back 2024-02-10T09:55:41.055732+0000 front 2024-02-10T09:55:41.055720+0000 (oldest deadline 2024-02-10T09:56:05.755786+0000) 2024-02-10T09:56:08.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:56:08.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:56:08.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:56:08.678Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:56:08.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:08 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:56:08.679Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:56:09.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:08 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:08.713+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:09.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:08 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:08.713+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:09.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:08 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:08.713+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:09.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:08 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:08.713+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6830 osd.3 since back 2024-02-10T09:55:41.055732+0000 front 2024-02-10T09:55:41.055720+0000 (oldest deadline 2024-02-10T09:56:05.755786+0000) 2024-02-10T09:56:10.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:09.701+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:10.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:09.701+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:10.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:09.701+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:10.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:09 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:09.701+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6830 osd.3 since back 2024-02-10T09:55:41.055732+0000 front 2024-02-10T09:55:41.055720+0000 (oldest deadline 2024-02-10T09:56:05.755786+0000) 2024-02-10T09:56:11.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:10 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:10.674+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:11.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:10 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:10.674+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:11.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:10 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:10.674+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:11.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:10 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:10.674+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6830 osd.3 since back 2024-02-10T09:55:41.055732+0000 front 2024-02-10T09:55:41.055720+0000 (oldest deadline 2024-02-10T09:56:05.755786+0000) 2024-02-10T09:56:12.081 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:11.691+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-02-10T09:55:19.353348+0000 front 2024-02-10T09:55:19.353394+0000 (oldest deadline 2024-02-10T09:55:44.653319+0000) 2024-02-10T09:56:12.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:11.691+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-02-10T09:55:24.654004+0000 front 2024-02-10T09:55:24.654026+0000 (oldest deadline 2024-02-10T09:55:49.953771+0000) 2024-02-10T09:56:12.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:11.691+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-02-10T09:55:34.054996+0000 front 2024-02-10T09:55:34.054865+0000 (oldest deadline 2024-02-10T09:55:55.754812+0000) 2024-02-10T09:56:12.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:11.691+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6830 osd.3 since back 2024-02-10T09:55:41.055732+0000 front 2024-02-10T09:55:41.055720+0000 (oldest deadline 2024-02-10T09:56:05.755786+0000) 2024-02-10T09:56:12.082 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:11 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7[68308]: debug 2024-02-10T09:56:11.691+0000 7f7ac22f6700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2024-02-10T09:55:47.357259+0000 front 2024-02-10T09:55:47.357225+0000 (oldest deadline 2024-02-10T09:56:10.857163+0000) 2024-02-10T09:56:12.831 INFO:journalctl@ceph.osd.7.smithi204.stdout:Feb 10 09:56:12 smithi204 bash[73422]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-osd-7 2024-02-10T09:56:14.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T09:56:14.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:56:14.307 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T09:56:14.307 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T09:56:14.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-09b43244-c7f5-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T09:56:14.581 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 systemd[1]: Stopping Ceph prometheus.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:56:14.582 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 bash[73660]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T09:56:14.953 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.662Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T09:56:14.954 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.663Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T09:56:14.954 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.663Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T09:56:14.954 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:14 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a[48594]: ts=2024-02-10T09:56:14.663Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T09:56:15.331 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Feb 10 09:56:15 smithi204 bash[73701]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T09:56:15.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09b43244-c7f5-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T09:56:15.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:56:15.479 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T09:56:15.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T09:56:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2024-02-10T09:56:15.613 INFO:teuthology.orchestra.run.smithi153.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2024-02-10T09:56:15.614 INFO:teuthology.orchestra.run.smithi153.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2024-02-10T09:56:15.614 INFO:teuthology.orchestra.run.smithi153.stderr: [--retry RETRY] 2024-02-10T09:56:15.614 INFO:teuthology.orchestra.run.smithi153.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install} 2024-02-10T09:56:15.614 INFO:teuthology.orchestra.run.smithi153.stderr: ... 2024-02-10T09:56:15.614 INFO:teuthology.orchestra.run.smithi153.stderr:cephadm: error: unrecognized arguments: --keep-logs 2024-02-10T09:56:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-10T09:56:15.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T09:56:15.744 INFO:teuthology.orchestra.run.smithi204.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2024-02-10T09:56:15.745 INFO:teuthology.orchestra.run.smithi204.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2024-02-10T09:56:15.745 INFO:teuthology.orchestra.run.smithi204.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2024-02-10T09:56:15.745 INFO:teuthology.orchestra.run.smithi204.stderr: [--retry RETRY] 2024-02-10T09:56:15.745 INFO:teuthology.orchestra.run.smithi204.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install} 2024-02-10T09:56:15.745 INFO:teuthology.orchestra.run.smithi204.stderr: ... 2024-02-10T09:56:15.745 INFO:teuthology.orchestra.run.smithi204.stderr:cephadm: error: unrecognized arguments: --keep-logs 2024-02-10T09:56:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-10T09:56:15.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:56:15.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:56:15.810 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T09:56:15.811 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/remote/ubuntu@smithi153.front.sepia.ceph.com/crash 2024-02-10T09:56:15.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T09:56:16.006 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/remote/ubuntu@smithi204.front.sepia.ceph.com/crash 2024-02-10T09:56:16.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T09:56:16.042 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T09:56:16.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T09:56:16.076 INFO:teuthology.orchestra.run.smithi153.stdout:2024-02-10T09:18:27.806179+0000 mon.a (mon.0) 153 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg inactive 2024-02-10T09:56:16.078 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T09:56:16.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[SEC\]' /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T09:56:16.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]' /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T09:56:16.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[WRN\]' /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T09:56:16.284 INFO:teuthology.orchestra.run.smithi153.stdout:2024-02-10T09:18:27.806179+0000 mon.a (mon.0) 153 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg inactive 2024-02-10T09:56:16.286 INFO:tasks.cephadm:Compressing logs... 2024-02-10T09:56:16.286 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T09:56:16.328 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T09:56:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T09:56:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-02-10T09:56:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T09:56:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T09:56:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T09:56:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T09:56:16.354 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T09:56:16.355 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-10T09:56:16.355 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T09:56:16.355 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T09:56:16.357 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T09:56:16.357 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T09:56:16.357 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.6.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T09:56:16.357 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-client.rgw.r.z.smithi204.kmhqqg.log 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.audit.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi153.stderr: 96.1% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.3.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.0.log: 91.8% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.audit.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-client.rgw.r.z.smithi204.kmhqqg.log: 71.5% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-client.rgw.r.z.smithi204.kmhqqg.log.gz 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.7.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mgr.x.log: 93.8% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T09:56:16.359 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T09:56:16.360 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T09:56:16.361 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T09:56:16.361 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.b.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T09:56:16.362 INFO:teuthology.orchestra.run.smithi204.stderr: 91.5% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T09:56:16.362 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T09:56:16.362 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T09:56:16.363 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T09:56:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.a.log: /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-client.rgw.r.z.smithi153.zyveqt.log 2024-02-10T09:56:16.367 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.5.log: 88.8% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T09:56:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-client.rgw.r.z.smithi153.zyveqt.log: 71.5% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-client.rgw.r.z.smithi153.zyveqt.log.gz 2024-02-10T09:56:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T09:56:16.375 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log: 93.7% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T09:56:16.375 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T09:56:16.383 INFO:teuthology.orchestra.run.smithi204.stderr: 94.7% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T09:56:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.1.log: 96.0% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T09:56:16.476 INFO:teuthology.orchestra.run.smithi204.stderr: 90.5% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T09:56:16.800 INFO:teuthology.orchestra.run.smithi153.stderr: 91.5% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T09:56:17.249 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T09:56:17.562 INFO:teuthology.orchestra.run.smithi204.stderr: 93.2% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T09:56:17.571 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T09:56:17.593 INFO:teuthology.orchestra.run.smithi153.stderr: 93.7% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T09:56:17.632 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T09:56:17.691 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T09:56:17.748 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T09:56:17.887 INFO:teuthology.orchestra.run.smithi204.stderr: 93.6% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T09:56:17.888 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-02-10T09:56:17.888 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m1.543s 2024-02-10T09:56:17.889 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m6.273s 2024-02-10T09:56:17.889 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.239s 2024-02-10T09:56:17.970 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T09:56:18.095 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T09:56:18.234 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% -- replaced with /var/log/ceph/09b43244-c7f5-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T09:56:18.236 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-02-10T09:56:18.236 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m1.893s 2024-02-10T09:56:18.236 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m9.118s 2024-02-10T09:56:18.236 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.329s 2024-02-10T09:56:18.236 INFO:tasks.cephadm:Archiving logs... 2024-02-10T09:56:18.236 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/remote/ubuntu@smithi153.front.sepia.ceph.com/log 2024-02-10T09:56:18.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T09:56:18.958 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:56:18.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:56:18.958 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:56:18.679Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:56:18.958 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:18 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:56:18.679Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:56:19.505 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/remote/ubuntu@smithi204.front.sepia.ceph.com/log 2024-02-10T09:56:19.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T09:56:20.414 INFO:tasks.cephadm:Removing cluster... 2024-02-10T09:56:20.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 --force 2024-02-10T09:56:21.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:56:21 smithi153 systemd[1]: Stopping Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:56:21.439 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:56:21 smithi153 bash[95853]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T09:56:22.189 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:56:21 smithi153 bash[95892]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T09:56:22.190 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:56:22 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T09:56:22.599 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:56:22 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T09:56:22.600 INFO:journalctl@ceph.node-exporter.a.smithi153.stdout:Feb 10 09:56:22 smithi153 systemd[1]: Stopped Ceph node-exporter.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:56:28.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=error ts=2024-02-10T09:56:28.679Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:56:28.939 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:56:28.679Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-02-10T09:56:28.940 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:28 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=warn ts=2024-02-10T09:56:28.680Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-02-10T09:56:33.808 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:33 smithi153 systemd[1]: Stopping Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:56:33.809 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:33 smithi153 bash[96168]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T09:56:34.098 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:33 smithi153 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a[60477]: level=info ts=2024-02-10T09:56:33.807Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T09:56:34.439 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:34 smithi153 bash[96207]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T09:56:34.832 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:34 smithi153 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-10T09:56:34.832 INFO:journalctl@ceph.alertmanager.a.smithi153.stdout:Feb 10 09:56:34 smithi153 systemd[1]: Stopped Ceph alertmanager.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:56:34.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09b43244-c7f5-11ee-95b8-87774f69a715 --force 2024-02-10T09:56:36.081 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:56:35 smithi204 systemd[1]: Stopping Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:56:36.082 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:56:35 smithi204 bash[74131]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T09:56:36.581 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:56:36 smithi204 bash[74172]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T09:56:37.012 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:56:36 smithi204 systemd[1]: Stopping Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:56:37.012 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:56:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T09:56:37.012 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:56:36 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T09:56:37.012 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Feb 10 09:56:36 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:56:37.331 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:56:37 smithi204 bash[74274]: Error: no container with name or ID "ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T09:56:37.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:56:37 smithi204 ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a[58968]: t=2024-02-10T09:56:37+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T09:56:37.831 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:56:37 smithi204 bash[74316]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T09:56:38.332 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:56:38 smithi204 systemd[1]: ceph-09b43244-c7f5-11ee-95b8-87774f69a715@grafana.a.service: Succeeded. 2024-02-10T09:56:38.336 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Feb 10 09:56:38 smithi204 systemd[1]: Stopped Ceph grafana.a for 09b43244-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:56:49.625 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T09:56:49.626 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T09:56:49.650 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T09:56:49.670 INFO:tasks.cephadm:Teardown complete 2024-02-10T09:56:49.670 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T09:56:49.681 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T09:56:49.682 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:56:49.693 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:56:49.711 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2024-02-10T09:56:49.717 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:56:49.717 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-02-10T09:56:49.717 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 -473us[ -473us] +/- 70ms 2024-02-10T09:56:49.717 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 7 377 39 -165us[ -189us] +/- 50ms 2024-02-10T09:56:49.717 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +830us[ +807us] +/- 83ms 2024-02-10T09:56:49.717 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:56:49.727 INFO:teuthology.orchestra.run.smithi204.stderr:bash: ntpq: command not found 2024-02-10T09:56:49.731 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:56:49.731 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-02-10T09:56:49.731 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 44 -444us[ -444us] +/- 70ms 2024-02-10T09:56:49.731 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 8 377 167 -186us[ -186us] +/- 48ms 2024-02-10T09:56:49.731 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +811us[ +811us] +/- 83ms 2024-02-10T09:56:49.731 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:56:49.732 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T09:56:49.743 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T09:56:49.744 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T09:56:49.764 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T09:56:49.838 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T09:56:49.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:56:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:56:49.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:56:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:56:49.965 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T09:56:49.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T09:56:49.997 INFO:teuthology.task.internal:Duration was 2992.479343 seconds 2024-02-10T09:56:49.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T09:56:50.033 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T09:56:50.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T09:56:50.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T09:56:50.081 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:56:50.082 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:56:50.449 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T09:56:50.449 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-02-10T09:56:50.449 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T09:56:50.472 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-02-10T09:56:50.472 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T09:56:50.495 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T09:56:50.495 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T09:56:50.515 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T09:56:50.618 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T09:56:50.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T09:56:50.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T09:56:50.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T09:56:50.877 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T09:56:50.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T09:56:50.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T09:56:50.932 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T09:56:50.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:56:50.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:56:50.971 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-02-10T09:56:51.001 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-02-10T09:56:51.048 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:56:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:56:51.066 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:56:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:56:51.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T09:56:51.116 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T09:56:51.117 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/remote/smithi153 2024-02-10T09:56:51.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T09:56:51.173 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369/remote/smithi204 2024-02-10T09:56:51.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T09:56:51.221 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T09:56:51.222 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T09:56:51.224 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T09:56:51.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T09:56:51.283 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T09:56:51.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T09:56:51.306 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T09:56:51.306 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T09:56:51.309 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T09:56:51.324 INFO:teuthology.orchestra.run.smithi153.stdout: 526076 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 09:56 /home/ubuntu/cephtest 2024-02-10T09:56:51.328 INFO:teuthology.orchestra.run.smithi204.stdout: 525634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 09:56 /home/ubuntu/cephtest 2024-02-10T09:56:51.329 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T09:56:51.340 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T09:56:51.415 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T09:56:51.445 DEBUG:teuthology.nuke:shortname: smithi153 2024-02-10T09:56:51.445 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:56:51.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:47.029875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMtRBaNzAsNSb86BThiggZUUAjJPohJ6V4KnVpQc+Lo04N4G5wJ15hooHO5/LKSP69dZL6FCHm/rhQRPDhnniQ='} 2024-02-10T09:56:51.473 DEBUG:teuthology.nuke:shortname: smithi204 2024-02-10T09:56:51.474 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:56:51.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:47.028807', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPThGYEcCYaN+tjd/oImrW4n0Is4gHrAHAeVaySmDdN+HDOZdhSvVh7LPgYKkYS/07ZgDwP79CZmJr2itqi1dD8='} 2024-02-10T09:56:51.522 INFO:teuthology.orchestra.console.smithi153:Power off 2024-02-10T09:56:51.522 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T09:56:51.540 INFO:teuthology.orchestra.console.smithi204:Power off 2024-02-10T09:56:51.541 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T09:56:51.549 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-02-10T09:56:51.550 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:56:51.565 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-02-10T09:56:51.565 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:56:51.575 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-02-10T09:56:51.592 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-02-10T09:56:55.577 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:56:55.592 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:56:55.690 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-02-10T09:56:55.705 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-02-10T09:56:59.692 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:56:59.705 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:56:59.805 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-02-10T09:56:59.818 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-02-10T09:56:59.906 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-02-10T09:56:59.919 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-02-10T09:57:00.023 INFO:teuthology.run:Summary data: description: rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} duration: 2992.479343175888 failure_reason: '"2024-02-10T09:18:27.806179+0000 mon.a (mon.0) 153 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg inactive" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T09:57:00.024 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T09:57:00.103 INFO:teuthology.run:FAIL